TABLE OF CONTENTS

TABLE OF CONTENTS
Table of contents ..............................................................................................1 Short abstract ...................................................................................................4 Extended abstract ............................................................................................5 Chapter 1 ........................................................................................................17 1.1 Multisensory integration ......................................................................17 1.2 Visual dominance.................................................................................20 1.3 The Colavita effect...............................................................................23 1.4 Early explorations of the Colavita effect .............................................25 1.4.1 Experimenter expectancy effects ..................................................25 1.4.2 Response requirements .................................................................26 1.4.3 Relative stimulus intensities .........................................................29 1.4.4 Arousal..........................................................................................31 1.4.5 Spatial confounds..........................................................................32 1.5 Early theories of the Colavita effect ....................................................33 1.6 Multisensory integration and the unity effect ......................................35 1.7 The Colavita effect and crossmodal extinction....................................37 1.8 Factors that may modulate the Colavita effect ....................................38 Chapter 2 ........................................................................................................40 2.0 Response requirements and response biases........................................40 2.1 Experiment 2.1.....................................................................................42 2.1.1 Methods.........................................................................................42 2.1.2 Results...........................................................................................45 2.1.3 Discussion .....................................................................................50 2.1.4 Sequence effects............................................................................52 2.1.5 Response coupling ........................................................................53 2.1.6 Response selection and the Colavita effect...................................57 2.2 Experiment 2.2.....................................................................................58 2.2.1 Methods.........................................................................................58 2.2.2 Results...........................................................................................59 2.2.3 Discussion .....................................................................................60 2.2.4 Between-participants analysis of Experiments 2.1 and 2.2 ..........63 2.2.5 Response biases ............................................................................63 2.3 General discussion ...............................................................................65 Chapter 3 ........................................................................................................68 3.0 The role of bimodal stimulus probability.............................................68 3.1 Experiment 3.1.....................................................................................70 3.1.1 Methods.........................................................................................70 3.1.2 Results...........................................................................................70 3.1.3 Discussion .....................................................................................72 3.1.4 Between-experiments analysis of Experiments 2.1 and 3.1 .........73 3.2 Experiment 3.2.....................................................................................74 1

TABLE OF CONTENTS

3.2.1 Methods.........................................................................................74 3.2.2 Results...........................................................................................75 3.2.3 Discussion .....................................................................................76 3.2.4 Between-experiments analysis of Experiments 3.1 and 3.2 .........77 3.3 Experiment 3.3.....................................................................................78 3.3.1 Methods.........................................................................................78 3.3.2 Results...........................................................................................79 3.3.3 Discussion .....................................................................................83 3.4 General discussion ...............................................................................83 Chapter 4 ........................................................................................................86 4.0 Attention and the Colavita effect .........................................................86 4.1 Experiment 4.1.....................................................................................95 4.1.1 Methods.........................................................................................95 4.1.2 Results...........................................................................................95 4.1.3 Discussion .....................................................................................99 4.1.4 Between-participants analysis of Experiments 2.1 and 4.1 ........101 4.1.5 Exogenous attention and the Colavita effect ..............................104 4.2 Experiment 4.2...................................................................................106 4.2.1 Methods.......................................................................................106 4.2.2 Results.........................................................................................108 4.2.3 Discussion ...................................................................................111 4.2.4 Between-participants analysis of Experiments 2.1 and 4.2 ........112 4.3 Experiment 4.3...................................................................................116 4.3.1 Methods.......................................................................................116 4.3.2 Results.........................................................................................116 4.3.3 Discussion ...................................................................................120 4.3.4 Between-participants analysis of Experiments 4.1 and 4.3 ........122 4.4 General discussion .............................................................................125 4.4.1 The law of prior entry .................................................................125 Chapter 5 ......................................................................................................128 5.0 Does audiovisual asychrony modulate the Colavita effect? ..............128 5.1 Experiment 5.1...................................................................................133 5.1.1 Methods.......................................................................................133 5.1.2 Results.........................................................................................135 5.1.3 Discussion ...................................................................................144 5.2 General discussion .............................................................................153 5.2.1 Prior entry ...................................................................................155 5.2.2 The unity effect ...........................................................................156 Chapter 6 ......................................................................................................158 6.0 Does spatial coincidence modulate the Colavita effect?....................158 6.1 Experiment 6.1...................................................................................162 6.1.1 Methods.......................................................................................162 6.1.2 Results.........................................................................................163 6.1.3 Discussion ...................................................................................167 6.2 General discussion .............................................................................168 2

TABLE OF CONTENTS

Chapter 7 ......................................................................................................172 7.0 Does semantic congruence modulate the Colavita effect? ................172 7.1 Experiment 7.1...................................................................................176 7.1.1 Methods.......................................................................................176 7.1.2 Results.........................................................................................178 7.1.3 Discussion ...................................................................................181 7.2 Experiment 7.2...................................................................................185 7.2.1 Methods.......................................................................................185 7.2.2 Results.........................................................................................186 7.2.3 Discussion ...................................................................................189 7.2.4 Dynamic stimuli..........................................................................191 7.3 General discussion .............................................................................193 7.3.1 Semantic congruency and the redundant target effect ................194 Chapter 8 ......................................................................................................199 8.0 A signal detection study of the Colavita effect..................................199 8.1 Experiment 8.1...................................................................................201 8.1.1 Methods.......................................................................................201 8.1.2 Results.........................................................................................203 8.1.3 Discussion ...................................................................................209 8.2 General discussion .............................................................................210 Chapter 9 ......................................................................................................216 9.1 Key findings.......................................................................................216 9.2 An explanation of the Colavita effect ................................................225 9.2.1 An explanation for how the visual stimulus dominates ..............226 9.2.2 When vision does not dominate in the Colavita task..................233 9.3 The Colavita effect & extinction........................................................237 9.3.1 Attention .....................................................................................239 9.3.2 Spatial and temporal separation ..................................................241 9.3.3 Perceptual and post-perceptual processes...................................242 9.4 Conclusions........................................................................................244 9.5 Suggestions for future research..........................................................244 Bibliography .................................................................................................247

3

SHORT ABSTRACT

SHORT ABSTRACT The Colavita effect provides a remarkable example of visual dominance,
demonstrating that a visual stimulus can dominate a person’s perception to such an extent that, on occasion, it appears to extinguish their perception of (or at least their ability to respond to) a concurrently-presented auditory stimulus. The aim of the experiments reported in this thesis was to investigate some of the key factors contributing to the Colavita visual dominance effect in order to gain a deeper understanding of its causes. The main findings reported in this thesis were that the Colavita visual dominance effect reflects a perceptuallybased phenomenon (see Chapters 2, 3, and 8), and is modulated by the modality to which participants endogenously and/or exogenously attend (Chapter 4), and the stimulus that they perceived first (Chapter 5). The spatial and temporal separation between the auditory and visual stimuli both appear to modulate the Colavita effect (Chapters 5 and 6), however, the semantic congruency between them does not appear to affect the magnitude of the Colavita effect that emerges (Chapter 7). An explanation is put forward that describes the Colavita effect as follows: Participants’ tendency to endogenously and/or exogenously direct their attention toward the visual modality results in the delayed perception of the auditory component of the bimodal stimulus. The auditory stimulus may therefore sometimes arrive at the primary stage of stimulus processing during the attentional dwell time of the visual stimulus. This may result in a difficulty in the consolidation and conscious report of the auditory stimulus because the processing of the visual stimulus temporarily occupies the participant’s limited-capacity attentional resources, thus resulting in the decay of the information concerning the auditory stimulus. The empirical findings reported in this thesis advance several important conclusions regarding the mechanism of the Colavita effect which, in turn, may contribute toward a greater understanding of the wider area of visual dominance, and could also provide a potential springboard for the study of crossmodal extinction in normal participants.

4

EXTENDED ABSTRACT

EXTENDED ABSTRACT
In Colavita’s (1974) original study, participants were instructed to press a ‘tone key’ whenever they heard a tone and a ‘light key’ whenever they saw a light. However, unbeknownst to the participants, a small number of bimodal trials (20% of the trials), in which both the light and tone were presented at the same time, were deliberately dispersed among the randomly alternating unimodal auditory and unimodal visual targets. In order to conceal the true nature of the experiment from his participants, Colavita originally deceived them by leading them to believe that the bimodal trials were ‘accidental’. Thus, the participants were presented with unimodal auditory, unimodal visual, and bimodal audiovisual stimuli and the point of interest was how they would respond on the occasional and unexpected bimodal trials. The surprising result to emerge from Colavita’s experiment was that the participants pressed the ‘light key’ on 98% of the trials and reported being totally unaware of the tone on a third of those trials. This striking example of the pre-potency of a visual stimulus over an auditory stimulus has come to be known as the Colavita visual dominance effect.

The Colavita effect (Colavita, 1974) represents one of the most startling examples of visual dominance in the psychology literature. It demonstrates that a visual stimulus can sometimes dominate our perception so much so that it can appear to extinguish the perception of (or people’s ability to respond to) a concurrentlypresented auditory stimulus. Since its initial discovery more than 30 years ago, the mechanisms underlying this remarkable phenomenon have remained something of an

5

EXTENDED ABSTRACT

enigma to researchers in the field. The primary aim of the research reported in this thesis was therefore to explore the factors contributing to the Colavita effect in order to try and develop a better understanding of the mechanisms underlying the effect. The hope being that the insights gained by investigating the causes of the Colavita effect might also provide a useful contribution to the broader area of visual dominance. The introductory chapter of this thesis provides a brief overview of the literature on multisensory integration, focusing specifically on research concerning the topic of visual dominance. The Colavita visual dominance effect is then introduced, and the early empirical studies and previous theories regarding the effect are outlined. The introductory chapter goes on to discuss a number of recent findings into the nature of audiovisual interactions which suggest that factors such as the modality toward which participants direct their attention (either endogenously or exogenously), and the assumption that participants have regarding the unity of the auditory and visual components of the bimodal stimulus (the extent to which participants may perceive them as constituting a unitary multisensory event rather than two separate unimodal events) may contribute to the Colavita effect. The contributions of these factors to the Colavita effect are explored in the following seven experimental chapters (Chapters 2-8). The experiments reported in Chapters 2 and 3 were designed to ascertain whether the Colavita effect reflects a genuine perceptual phenomenon (rather than being caused by response biases, the particular instructions given to participants in terms of how they should respond to bimodal targets, etc). Chapter 4 explores the contribution of attention, either directed endogenously or exogenously toward audition or vision, to the Colavita effect. Meanwhile, the experiments reported in

6

EXTENDED ABSTRACT

Chapters 5, 6, and 7 investigate whether the spatial separation, temporal separation, and semantic congruency between the auditory and visual stimuli, respectively, can also modulate the magnitude of the Colavita effect. Finally, the experiment reported in Chapter 8 explores the differential contributions of perceptual and post-perceptual processes to the Colavita effect. In the General Discussion (Chapter 9), the main findings of the experimental chapters are summarized, and an explanation describing how the Colavita effect may emerge is put forward. The basic methodology for investigating the Colavita effect in the experiments reported throughout this thesis is as follows. Participants were presented with blocks of trials consisting of unimodal auditory, unimodal visual, and bimodal targets in the ratio of 40A:40V:20AV (with an interstimulus interval ranging between 1450ms1750ms; note that the relative probability of the three types of target was manipulated in the experiments reported in Chapter 3). The participants were instructed to discriminate whether the target was a unimodal auditory, unimodal visual, or bimodal target as quickly as possible. The participants were instructed to press the auditory response key whenever a unimodal auditory stimulus was presented, to press the visual response key in response to a unimodal visual stimulus, and both response keys whenever a bimodal stimulus was presented1 (note, though, that a different set of instructions as to how to respond to the stimuli was given in Experiment 2.2, which will be discussed later). The Colavita effect is defined as occurring when the percentage of bimodal trials in which the participants made visual-only responses (i.e., when the participant responded to the bimodal stimulus as if only a unimodal visual stimulus had been presented) is greater than the percentage of auditory-only

Participants were given no specific instructions with regard to whether or not they should respond simultaneously to the two targets.

1

7

EXTENDED ABSTRACT

responses. The magnitude of the Colavita effect is calculated as the difference between the percentage of visual-only and auditory-only responses on the bimodal trials. Different aspects of this basic methodology were manipulated in the experiments reported in this thesis in order to investigate the influence of various factors (e.g., the instructions given to the participants as to how to respond to the three types of target, the spatial or temporal coincidence between the auditory and visual components of the bimodal stimulus, the relative frequency with which the auditory, visual, and bimodal targets were presented, etc) on the magnitude of the Colavita effect. There are a number of problematic factors that make any straightforward interpretation of the early studies of the Colavita effect rather difficult (factors such as deceit by the experimenter regarding the presentation of the bimodal targets, the presentation of auditory and visual stimuli from different spatial positions, difficulties in interpreting the participants’ responses, and the relatively infrequent presentation of bimodal targets). Therefore, the aims of the experiments reported in Chapters 2 and 3 were firstly, to find the optimal methodology with which to investigate the Colavita effect (i.e., a methodology that is not subject to the previously discussed confounds), and secondly, to ascertain whether the effect reflects a genuine perceptual phenomenon (i.e., the Colavita effect occurs when those factors such as response biases, experimenter deceit, etc, have been controlled for). Thus, the participants in the experiments reported in Chapters 2 and 3 were presented with a version of the Colavita task that was free from the aforementioned confounds, in which the response requirements of the task (i.e., how the participants were instructed to respond to the unimodal auditory, unimodal visual, and bimodal targets; Chapter 2) or the

8

EXTENDED ABSTRACT

probability with which the bimodal targets were presented (10%, 33%, 50%, 60%, or 90% of the trials; Chapter 3) were manipulated. In Chapter 2, the participants were instructed with two different types of response requirement: the two-key response condition (Experiment 2.1) and the threekey response condition (Experiment 2.2; cf. Sinnett et al., 2007). In both the two-key and three-key response conditions, the participants were instructed to press the auditory response key in response to a unimodal auditory stimulus, and the visual response key in response to a unimodal visual stimulus. Participants in the two-key response condition were clearly instructed to press both the auditory and visual response keys in response to a bimodal stimulus (i.e., this was the same set of response requirements as those that were used in the majority of experiments reported in this thesis), whereas in the three-key response condition, the participants were instructed to press a third, separate bimodal response key instead. These response requirements have the advantage that they did not require the participants to explicitly decide which of the two components to respond to2, which could potentially introduce response biases into the results (i.e., if participants were more biased to respond to the visual stimulus, the Colavita visual dominance effect would appear to be larger due to this bias). The main finding to emerge from the experiments reported in Chapter 2 was that the Colavita effect manifests itself in both the two-key and three-key response requirement conditions, thus suggesting that the Colavita effect is not simply an artefact of the response requirements of the particular task given to the participants. Furthermore, the results of the two experiments reported in Chapter 2 also ruled out an explanation of the Colavita effect in terms of response bias, deceit by the

This contrasts with some of the early studies of the Colavita effect in which the participants were explicitly instructed to respond to the signal that they perceived first (Colavita, 1974; Colavita et al., 1976; Egeth & Sager, 1977; Johnson & Shapiro, 1989; Quinlan, 2000; Shapiro et al., 1984).

2

9

EXTENDED ABSTRACT

experimenter, the presentation of auditory and visual stimuli from different spatial positions, or difficulties in interpreting the participants’ responses. The experiments reported in Chapter 3 investigated whether the Colavita effect (and the difficulty that participants seemingly have in responding to bimodal targets) emerges solely due to the relatively infrequency with which bimodal targets are typically presented (i.e., 20% of the trials). To address this question, the relative probabilities of unimodal auditory, unimodal visual, and bimodal targets were varied; the three types of target were presented in the ratios of: 45A:45V:10AV, 33A:33V:33AV, 25A:25V:50B, 20A:20V:60AV, or 5A:5V:90AV in the experiments reported in Chapter 3. The principal finding to emerge from those experiments was that the Colavita effect can be observed even when the bimodal targets are not presented any less frequently than the unimodal targets, therefore suggesting that the emergence of the Colavita effect (and the apparent difficulty that participants have in responding to bimodal targets) is not attributable solely to the low frequency with which the bimodal targets were presented in the majority of previous studies. To summarize, the results reported in Chapters 2 and 3 provide support for the argument that the Colavita effect represents a perceptually-based phenomenon (i.e., the Colavita effect is not simply caused by any bias that participants may have to respond to the visual stimulus, the particular response requirements of the task, or the relatively low frequency with which the bimodal targets were presented). Having established that the Colavita effect is indeed a perceptually-based phenomenon, the aim of the experiments reported in the following chapters was to investigate some of those factors that modulate the effect. In Chapter 4, the contribution of attention toward the emergence of the Colavita effect was evaluated. For many years, researchers have suggested that people

10

EXTENDED ABSTRACT

have a tendency to direct their attention endogenously toward the visual modality (e.g., Battaglia et al., 2003; Posner et al., 1976; Spence et al., 2001a), and early researchers of the Colavita effect suggested that the effect arises from the tendency that participants have to endogenously attend toward vision (see Colavita & Weisberg, 1979; Egeth & Sager, 1977). Another factor proposed in Chapter 4 that could potentially contribute to the Colavita effect is the greater capacity of visual (than auditory) stimuli to capture a person’s attention exogenously (i.e., attention is drawn away from the auditory stimulus as a result of having been attracted exogenously toward the visual stimulus; e.g., Hamlin, 1895; Rodway, 2005; Smith, 1933; Spence et al., 2001a; Turatto et al., 2002). According to this potential explanation, a participant’s failure to respond to the auditory component of a bimodal stimulus might be caused by their attention being captured exogenously by the visual component of the bimodal stimulus (and away from the auditory stimulus). Thus, the three experiments reported in Chapter 4 investigated the influence of attention (directed endogenously, exogenously, or both endogenously and exogenously, toward audition and/or vision) on the Colavita effect. The modality toward which a participant’s attention was directed endogenously was manipulated by instructing them to expect either auditory or visual stimuli throughout a whole block of experimental trials (Experiments 4.1 and 4.3). Meanwhile, the modality toward which participants’ attention was directed exogenously was manipulated by varying the modality of the exogenous cues which preceded the onset of the target stimulus (note, an exogenous cue preceded the target on every trial; Experiments 4.2 and 4.3). The main finding to emerge from the three experiments reported in Chapter 4 was that the magnitude of the Colavita effect was significantly affected by both the endogenous and exogenous attentional manipulations. In particular, the results of the

11

EXTENDED ABSTRACT

experiments reported in Chapter 4 provided support for the notion that people do tend to attend endogenously toward the visual modality, and suggest that this tendency contributes to the emergence of the Colavita effect. The results also support the idea that visual stimuli have a greater capacity than auditory stimuli to capture attention exogenously, and that this may also be a factor contributing to the Colavita effect. The modality toward which participants attend endogenously, however, appears to have a stronger effect on the magnitude of the Colavita effect than the modality toward which participants attend exogenously; the Colavita effect emerged when participants endogenously attended toward vision regardless of whether the preceding exogenous cue had been auditory or visual, and the auditory dominance that emerged when participants endogenously attended toward audition was attenuated, but not reversed, when exogenous visual cues were presented. The explanation put forward in Chapter 4 for how attention modulates the Colavita effect was in terms of the law of prior entry (Spence et al., 2001b; Titchener, 1908). According to the law of prior entry, attended stimuli are perceived as having been presented earlier in time than are simultaneously-presented stimuli that are not attended (see James, 1890; Mollon & Perkins, 1996; Titchener, 1908). Thus, participants’ tendency to have their attention directed (either endogenously or exogenously) toward the visual modality would, according to the law of prior entry, result in their perceiving the visual component of the bimodal target prior to the auditory component (or, in other words, participants would have a delayed perception of the auditory stimulus). One of the aims of the experiment reported in Chapter 5 was therefore to determine whether the prior entry of the visual stimulus (or the delayed perception of the auditory stimulus) might somehow contribute to the Colavita effect. If the delayed perception of the auditory stimulus contributes to the Colavita effect,

12

EXTENDED ABSTRACT

then it seemed likely that the magnitude of the Colavita effect should increase when the visual stimulus is presented first, and be attenuated or even reversed when the auditory stimulus is presented first. The experiment reported in Chapter 5 investigated whether the Colavita effect is modulated by the temporal order in which auditory and visual stimuli are presented and the temporal separation between them. This question was explored by manipulating the order in which the auditory and visual stimuli were presented, and the temporal interval between them. Two main findings emerged from the experiment reported in Chapter 5: First, the Colavita effect was modulated by the temporal order in which the asynchronously-presented auditory and visual components of the bimodal stimulus were presented. As predicted by the prior entry explanation of the Colavita effect (discussed earlier), participants having a delayed perception of the auditory component of the bimodal stimulus (presumably due to their attending to vision) could potentially contribute to the Colavita effect: the Colavita effect was larger when the visual stimulus preceded the auditory stimulus, and was reversed/attenuated when the auditory stimulus led. Second, the Colavita effect is also modulated by the temporal separation between the auditory and visual components of the bimodal stimulus; in particular, the Colavita effect was eliminated when the temporal separation between the auditory and visual stimuli was large enough that participants could respond to them as if they were two sequentially-presented unimodal stimuli. The magnitude of the Colavita effect did not, however, appear to be modulated by the extent to which participants temporally bound the auditory and visual stimuli into a unitary percept. One of the aims of the experiments reported in Chapters 5-7 was to ascertain whether the factors that influence the extent to which participants perceive auditory

13

EXTENDED ABSTRACT

and visual stimuli as constituting a single unitary event (that is, the assumption of unity between them) could modulate the magnitude of the Colavita effect. According to the ‘unity effect’, the stronger an observer’s assumption that two events refer to the same unimodal object rather than to two separate events, the greater the intersensory bias, or intersensory binding, between them (Bedford, 2001; Welch & Warren, 1980; see Spence, 2007, for a recent review). It therefore follows that the magnitude of the Colavita effect should be modulated by the extent to which participants perceive the auditory and visual stimuli as belonging to the same event versus when they perceive them as belonging to separate, independent events. The binding versus segregation of unimodal auditory and visual stimuli has been shown to depend on their spatial and temporal separation (for reviews, see Calvert et al., 2004; Vatakis & Spence, 2007), and has even been shown to be modulated by their semantic congruency under certain conditions (see Laurienti et al., 2004; Molholm et al., 2004; Taylor et al., 2006; Vatakis & Spence, 2007b). The influence of temporal separation, spatial separation, and semantic congruency on the magnitude of the Colavita effect were therefore investigated in the experiments reported in Chapters 5, 6, and 7 (i.e., by varying the temporal separation, spatial separation, and semantic congruency, respectively, between the auditory and visual stimuli). The results of the experiments reported in Chapters 5 through 7 revealed that the temporal and spatial separation between the auditory and visual components of a bimodal stimulus can modulate the Colavita effect, but the semantic congruency between the auditory and visual stimuli does not. The effect of temporal separation on the magnitude of the Colavita effect was explained in terms of the Colavita effect being attenuated (or eliminated) when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli.

14

EXTENDED ABSTRACT

Meanwhile, the effect of spatial separation on the magnitude of the Colavita effect was explained in terms of the Colavita effect being attenuated (or eliminated) when participants have a redundant spatial cue informing them that two stimuli had been presented. In contrast to most of the chapters in the thesis which focussed on determining the factors that contribute to the Colavita visual dominance effect, the experiment reported in Chapter 8 explored the processes involved in the effect. Signal detection theory (SDT) was used as a means of quantitatively evaluating the relative contributions of perceptual and decisional processes to the Colavita effect. The participants in Experiment 8.1 were therefore presented with a version of the Colavita task that had been modified so that SDT analyses could be performed on the data. The main result to emerge from Experiment 8.1 was that participants exhibited a significant decrease in sensitivity (but no significant change in their response criterion) to auditory stimuli when they were presented together with visual stimuli. In other words, the emergence of the Colavita visual dominance effect (at least for the effect that emerged in Experiment 8.1) was caused by participants finding it difficult to perceptually process the auditory stimulus, rather than being caused by participants having a bias to respond to the visual stimulus (a decisional process). Thus, the Colavita effect that emerged in Experiment 8.1 appeared to reflect a perceptual, rather than a decisional, process. The General Discussion (Chapter 9) integrates the findings from all of the experiments reported in this thesis in order to generate a plausible explanation to describe how the Colavita effect occurs (i.e., how the visual component of a bimodal stimulus can preclude the perception of the auditory component). The final chapter concludes with a more general discussion of the contribution of the findings reported

15

EXTENDED ABSTRACT

in this thesis to the field of multisensory integration. The further study of the Colavita effect is also put forward as a potential springboard toward a better understanding of crossmodal extinction in normal participants. Finally, a number of potential directions for future research which would advance the progress made in, and increase the value of, researching the Colavita effect are proposed.

16

CHAPTER 1 - INTRODUCTION

CHAPTER 1
1.1 MULTISENSORY INTEGRATION “All we have to believe with is our senses, the tools we use to perceive the world: our sight, our touch, our memory. If they lie to us, then nothing can be trusted. And even if we do not believe, then still we cannot travel in any other way than the road our senses show us; and we must walk that road to the end.”

- Neil Gaiman, 2001, American Gods

Our experience and awareness of the vibrant world that surrounds us is the result of the perception derived from our many senses (e.g., sight, hearing, smell, taste, and touch). From the beginnings of early Greek philosophy, at the time of Aristotle (see Aristotle’s, De Anima, 1977), to the contemplative meditation in Tibetan Buddhist temples, to the modern day psychology laboratory; scientists, philosophers, and great thinkers alike have all pondered about how individuals achieve a perception of the world around them. Significant advances have been made in discovering the mechanisms by which information is transformed within each sensory system, which is one step toward understanding how individuals perceive their environment. Our everyday perceptions should not, however, be considered as just parallel and independent experiences of incidents occurring in separate sensory modalities. 17

CHAPTER 1 - INTRODUCTION

Rather, our perceptual experiences are formed by the integration of information from the various senses. Indeed, the focus of an increasing number of researchers is now turning toward discovering how the different senses work together in concert (e.g., see the chapters in Calvert, Spence, & Stein, 2004; Spence & Driver, 2004), and a growing body of research now supports the idea that our brains integrate the information from the various sensory channels in such a way as to enhance the perception and reaction to events in the external world (e.g., Amedi, Malach, Hendler, Peled, & Zohary, 2001; Calvert, Bullmore, Brammer, Campbell, Williams, McGuire, Woodruff, Iversen, & David, 1997; Macaluso, Frith, & Driver, 2000; also see Spence & McDonald, 2004, for a review).

Figure 1.1. We integrate information from multiple sensory modalities in many aspects of our daily lives; for example, whilst driving we combine the auditory warning signals with the visual information in front of us (e.g., Chan & Chan, 2006), we combine visual, gustatory, and olfactory information when eating (see Small, 2004, for a review), and we integrate auditory and visual speech signals when watching television (Basil, 1994; ITU-R BT.1359-1, 1998; Reeves & Voelker, 1993; Rihs, 1995).

18

CHAPTER 1 - INTRODUCTION

At present, one of the most extensively studied sensory interactions in the field of multisensory research is the interaction between the auditory and visual modalities (e.g., see Thesen, Vibell, Calvert, & Österbauer, 2004). There are many examples of audiovisual interactions in the literature; for example, researchers have investigated the auditory and visual contributions to speech (e.g., Dixon & Spitz, 1980; McGurk & MacDonald, 1976; Munhall, Gribble, Sacco, & Ward, 1996; Sumby & Pollack, 1954; Vatakis & Spence, 2007a), to multisensory object recognition (e.g., Giard & Peronnet, 1999; Laurienti, Kraft, Maldjian, Burdette, & Wallace, 2004; Molholm, Ritter, Javitt, & Foxe, 2004; Taylor, Moss, Stamatakis, & Tyler, 2006; see Amedi, von Kriegstein, van Atteveldt, Beauchamp, & Naumer, 2005, for a review), to the localization of objects (e.g., Bertelson & Aschersleben, 1998; Howard & Templeton, 1966; Pick, Warren, & Hay, 1969; see Vroomen & de Gelder, 2004, for a review), and to the effectiveness of multisensory warning signals in cars (see Ho, Reed, Spence, 2007; Ho & Spence, 2005, for a review). As information from the auditory and visual sensory modalities often contains redundant information about events in the external world (Knudsen & Brainard, 1995), people tend to integrate the audiovisual signals into a multisensory percept in order to optimise their perception of, and responses toward, events in the external world (see Calvert et al., 2004). A greater understanding of the mechanisms underlying the audiovisual interactions involved in perception is therefore of great importance, not only to the academic field of multisensory research, but also because of its potentially farreaching practical applications (e.g., Chan & Chan, 2006; Spence & Driver, 1997c). For example, multimodal user interfaces and media (such as driving interfaces, computer-, internet-, or television- based media, computer games, etc.) all involve the integration of auditory and visual information to maximise the ability of the user to

19

CHAPTER 1 - INTRODUCTION

absorb, enjoy, and/or react to the information. The aim of the experiments reported in this thesis will be to investigate one of the most widely researched facets of audiovisual interactions, namely the dominance of vision over audition. Specifically, the thesis will explore the factors contributing to the Colavita visual dominance effect (Colavita, 1974), a particularly fascinating example of visual dominance. The insights gained by exploring the causes of the Colavita effect, will hopefully contribute toward a greater understanding of the processes involved in visual dominance in general, which will in turn enrich the field of audiovisual integration.

1.2

VISUAL DOMINANCE “Seeing is believing”
One aspect of multisensory integration that has been of enduring interest to

researchers is the question of how the contributions of auditory and visual information are weighted in order to produce the final multisensory percept (e.g., see Ernst & Banks, 2002). Researchers have argued that when auditory and visual stimuli are presented to participants the most pertinent (or accurate) sensory modality for the task tends to dominate (especially in situations when the information from the stimuli conflicts; see Freids, 1974; Rock & Harris, 1964; Welch & Warren, 1980, 1986, for reviews; see Ernst & Banks, 2002; Ernst & Bülthoff, 2004, for a model of how inherently noisy information from different sensory modalities is weighted to optimise the accuracy of the estimate). In general, audition tends to dominate over vision in the temporal domain (e.g., Fendrich & Corballis, 2001; Morein-Zamir, Soto-Faraco, & Kingstone, 2003; Recanzone, 2003; Shams, Kamitami, & Shimojo, 2000; Shimojo & Shams, 2001; Vroomen & Keetels, 2006; Watanabe & Shimojo, 1998; Welch, 20

CHAPTER 1 - INTRODUCTION

DuttonHurt, & Warren, 1986; though see Shams, Kamitami, & Shimojo, 2004, for examples when audition dominates over vision outside the temporal domain), and in situations of aversive motivation (e.g., Shapiro, Egerman, & Klein, 1984). However, the pervasive pattern of dominance that has been seen most frequently throughout the multisensory literature (at least in the psychologist’s laboratory) is that of visual dominance over audition (see Figure 1.2).
1200

1000

Number of articles

800

600

400

200

0 Auditory dominance Visual dominance

Figure 1.2. Graph showing the number of articles on Google Scholar with the th terms ‘auditory dominance’ and ‘visual dominance’ in the abstract or title (17 May, 2007).

For example, presenting synchronous auditory and visual signals from slightly different spatial locations can create the illusion that the sound is coming from the direction of the visual stimulus, in the ventriloquist effect (Bertelson, 1998; Bertelson & Aschersleben, 1998; Bertelson & Radeau, 1976, 1981; Howard & Templeton, 1966; Jackson, 1953; Pick et al., 1969; see Bertelson, Vroomen, de Gelder, & Driver, 2000, for a review). In the McGurk effect, an incongruent visual speech signal alters people’s perception of the auditory speech signal (McGurk & MacDonald, 1976; Munhall et al., 1996). For dynamic stimuli, visual motion signals can influence the perception of auditory motion signals (but not vice versa; Allen & Kolers, 1981; Anstis, 1973; Kitagawa & Ichihara, 2001). Visual dominance over audition has also been demonstrated in animals (Partan & Marler, 1999), including cows (Uetake & 21

CHAPTER 1 - INTRODUCTION

Kudo, 1994), pigeons (Foree & LoLordo, 1973; Randich, Klein, & LoLordo, 1978; Shapiro, Jacobs, & LoLordo, 1980), and rats (Meltzer & Masaki, 1973). There are many more examples of visual dominance over audition in the literature (for reviews, see Bertelson & de Gelder, 2004; Posner, Nissen, & Klein, 1976; see also the chapters in Calvert et al., 2004). However, one of the most fascinating (if little studied) instances of visual dominance has been provided by research on the Colavita visual dominance effect (Colavita, 1974), which will be the focus of this thesis. The Colavita effect is a phenomenon whereby the presentation of a visual stimulus appears to extinguish people’s perception of, or ability to respond to, a concurrently-presented auditory stimulus (see the following section for a detailed description). Research into the Colavita effect was primarily conducted in the late 1970s and early 1980s by Francis Colavita and his colleagues at the University of Pittsburgh (Colavita, 1974; Colavita, Tomko, & Weisberg, 1976; Colavita & Weisberg, 1979), as well as others such as Egeth and Sager (1977), and Shapiro and his colleagues (Johnson & Shapiro, 1989; Shapiro et al., 1984), but has since been mentioned only sporadically in the literature as an example of visual dominance. One of the reasons for this may have been the poor experimental control which plagued the early studies, perhaps leading later researchers who came across the phenomenon to believe that the effect reflected nothing more than a response bias. Recently, however, there has been something of a renewed spate of interest and research into the Colavita visual dominance effect (e.g., Sinnett, Spence, & SotoFaraco, 2007; Quinlan, 2000). This has most likely been fuelled by the recent realisation, by at least some researchers, of the value of studying the Colavita effect, in terms of its potential in contributing to a greater understanding of the mechanisms involved in visual dominance. The following sections of this chapter describe the

22

CHAPTER 1 - INTRODUCTION

Colavita effect, detail the research that followed on from Colavita’s original study, list some of the potential future applications of research on the Colavita effect, and finally the factors that may modulate the Colavita effect will be discussed.

1.3

THE COLAVITA EFFECT
In Colavita’s (1974, Experiment 1) original study, the participants were

instructed to press a ‘tone key’ whenever they heard a tone and a ‘light key’ as soon as they saw a light. However, unbeknownst to the participants, a small number of bimodal trials, in which both the light and tone were presented at the same time, were deliberately dispersed among the randomly alternating unimodal auditory and unimodal visual targets (5 out of the 35 trials presented to each of the 10 participants). In order to conceal the true nature of the experiment from his participants, Colavita originally deceived them into believing that the bimodal trials were ‘accidental’1. That is, during the practice trials, Colavita claimed to have ‘accidentally’ presented both stimuli at once, drew his participant’s attention to what had just happened, apologised, and then continued to demonstrate how such an accident could have resulted from the experimenter’s failure to open a switch on the preceding trial. Thus, the participants were presented with unimodal auditory, unimodal visual, and bimodal audiovisual stimuli and the point of interest was how they would respond on the bimodal trials. The compelling result that emerged from this experiment was that the participants pressed the light key on 49 out of the 50 trials (across all 10 participants). Moreover, the participants in Colavita’s (1974) study actually reported being totally unaware of the tone on 16 of those 49 trials (and on the 1 trial when the tone key was
Note that the participants were therefore given no instruction as to how to respond on bimodal trials, nor were they even informed that such bimodal trials might occur again.
1

23

CHAPTER 1 - INTRODUCTION

pressed, the participant reported that he had made a mistake and that he should have pressed the light key instead). These results led Colavita to report that “in one manner or another, the visual stimulus was prepotent over the auditory stimulus on all 50 of the conflict [i.e., bimodal] trials” (Colavita, 1974, p. 410). While Colavita and his colleagues (Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979) defined the ‘prepotency’ of the visual over the auditory stimulus in terms of the fact that participants made more light key than tone key responses on the bimodal trials, subsequent research has modified the definition so that the Colavita effect can be determined quantitatively (e.g., Sinnett et al., 2007; also see Shapiro & Johnson, 1989). The Colavita effect is now typically defined as occurring when participants make significantly more visual-only errors than auditoryonly errors (see Sinnett et al., 2007). A visual-only error occurs when participants respond to the bimodal stimulus as if it was a unimodal visual stimulus (i.e., when they press only the light key on a bimodal trial), and an auditory-only error occurs when participants respond to the bimodal stimulus as if it was a unimodal auditory stimulus (i.e., when they press only the auditory key on bimodal trials). The magnitude of the Colavita effect reflects the difference between the percentage of visual-only and auditory-only errors made by the participants on the bimodal trials. This definition of the Colavita effect will be used throughout the thesis because it provides the best measure of the Colavita effect currently available in the literature (in terms of not being subject to the response bias explanations, and other confounds, that previous definitions of the Colavita effect have been subject to; as will be discussed in Section 1.4.2), and it is the closest to Colavita’s (1974) original definition. Following Colavita’s (1974) striking finding, a number of researchers investigated the phenomenon over the next decade and a half (Colavita et al., 1976;

24

CHAPTER 1 - INTRODUCTION

Colavita & Weisberg, 1979; Egeth & Sager, 1977; Johnson & Shapiro, 1989; Shapiro et al., 1984). They investigated the contribution of a number of factors to the Colavita effect, such as experimenter expectancy effects, the effect of the response requirements of the task, the relative intensities of the auditory and visual stimuli (see the following Section, 1.4.3), and the modality toward which participants directed their attention endogenously (or voluntarily; see Section 1.5). However, a few factors still remain to be investigated (e.g., relating to the response requirements of the task, and the relative spatial positions from which the auditory and visual stimuli were presented; see Sections 1.4.2 and 1.4.5), and these will be detailed in the next section.

1.4

EARLY EXPLORATIONS OF THE COLAVITA EFFECT

1.4.1 Experimenter expectancy effects The fact that participants in Colavita’s (1974, Experiment) original study were not informed that bimodal stimuli might be presented, and were consequently not instructed as to how to respond on such trials, makes it possible that experimenter expectancy effects may have contributed to the visual dominance observed. In order to eliminate such effects arising from this ‘deceit’ (e.g., Colavita, 1974, Experiments 1 & 2; Colavita & Weisberg, 1979), participants were thereafter informed that bimodal stimuli would be presented and were furthermore instructed as to how to respond to such stimuli (see Colavita 1974, Experiments 3 & 4; Colavita et al. 1976; Egeth & Sager, 1977; Johnson & Shapiro, 1989; Quinlan, 2000; Shapiro et al., 1984; Sinnett et al., 2007). Nevertheless, the participants still failed to respond to the sound on the majority of the bimodal trials even when the use of deception was discontinued, and even when the participants were explicitly instructed as to how to

25

CHAPTER 1 - INTRODUCTION

respond on the bimodal trials. It should, however, be noted that the Colavita effect observed in the more recent studies of the phenomenon has typically been much smaller in magnitude than that originally observed by Colavita, possibly due to the removal of other certain factors (as discussed below).

1.4.2 Response requirements In all of the studies of the Colavita effect that have been reported to date, the participants were required to press the tone key in response to tones, and the light key in response to lights. In an attempt to investigate whether the Colavita effect might simply reflect the response demands of the task, a number of subsequent studies used a variety of different response requirements for participants to indicate the presence of a bimodal target (see Table 1.1 for a summary of the instructions given, and the visual dominance observed, in the early studies of the Colavita effect). Yet, in most cases, a robust Colavita effect was still observed. For example, in certain studies, the participants were instructed to press the key corresponding to the stimulus (i.e., auditory or visual) that they detected first on the bimodal trials (e.g., Colavita, 1974, Experiment 3; Colavita et al., 1976; Egeth & Sager, 1977, Experiments 3, 5, & 6; Johnson & Shapiro, 1989; Shapiro et al., 1984). It should be noted that under such conditions, however, responses would not necessarily be indicative of participants’ awareness of the stimuli, but may simply have reflected any biases in response selection participants may have had when they were forced to choose which of the two stimuli they wanted to respond to.

26

CHAPTER 1 - INTRODUCTION

Table 1.1. Table summarizing the response requirements and the visual dominance effects observed in previous studies of the Colavita effect. Visualonly responses occurred when participants responded to the bimodal stimulus by pressing only the visual response key. Vision-first responses refer to trials in which participants responded to the visual component of the bimodal stimulus first, or when they perceived the visual component first. A high percentage of visual-only or vision-first responses was taken to indicate visual dominance. Egeth and Sager (1977) interpreted the slowing of responses (reaction times: RTs) to bimodal stimuli (a tone key press) relative to responses to the unimodal auditory stimuli as indicating that the visual stimulus interfered with (i.e., dominated over) the processing of the auditory stimulus (see Section 1.4.2 for a detailed explanation).
Study Colavita, 1974 Experiment 1 Visual dominance effect Response requirements

98% visual-only responses

Participants were instructed to press the tone key as soon as they saw a tone, and the light key as soon as they saw the light. They were not informed that bimodal stimuli would be presented, or instructed as to how to respond to them. Same as Colavita (1974, Experiment 1). Participants were instructed to press the tone key as soon as they saw a tone, and the light key as soon as they saw the light. On bimodal trials, participants were instructed to press the key corresponding to the signal they perceived first. Same as Colavita (1974, Experiment 3), except that on bimodal trials, participants were instructed to press the tone key.

Experiment 2 Experiment 3

87% visual-only responses 94% visual-first responses

Experiment 4

60% visual- first responses

Colavita et al., 1976 Experiment 1 Experiment 2 Colavita & Weisberg, 1979

84% visual- first responses 76% visual- first responses 98% visual-only responses

Same as Colavita (1974, Experiment 3). Same as Colavita (1974, Experiment 3). Same as Colavita (1974, Experiment 1). Note that participants were now responding to the offset of the stimuli because Colavita and Weisberg believed that stimulus offset might not elicit as much of an orienting response as a stimulus onset (cf. Sokolov, 1963). Hence, they reasoned that if the Colavita effect occurred even when a supposedly less alerting stimulus offset was used, then the phenomenon must be due to post-perceptual processes. Participants were instructed to press the tone key as soon as they saw a tone, and the light key as soon as they saw the light. On bimodal trials, participants were instructed to press both keys, but were not given any instructions about the order in which the keys were to be pressed. Note that in this experiment (but not in subsequent experiments in the Egeth & Sager, 1977, study) the stimuli were presented continuously until the corresponding response was made. Participants were informed that unimodal auditory, unimodal visual, and bimodal stimuli would be presented. They were instructed to press the tone key whenever there was a tone (i.e., regardless of whether or not there was a light), and to press nothing when there was a light. Same as Colavita (1974, Experiment 4) except that the ratio of unimodal visual trials and bimodal trials was varied.

Egeth & Sager, 1977 Experiment 1

76% vision-first responses

Experiment 2

The bimodal RTs (347ms) were not significantly slower than unimodal auditory RTs (375ms) (no visual dominance). In conditions where there were more bimodal than unimodal visual trials, the bimodal RTs were significantly slower than unimodal auditory RTs (visual dominance).

Experiments 3-6

Shapiro et al., 1984 Experiment 1

76% vision-first responses in the noshock condition, 38% vision-first responses in the shock condition

Same as Colavita (1974, Experiment 3). There were two arousal conditions (shock or no-shock). In the shock conditions, participants were informed that shocks would occur on random trials and they were shocked on 20% of the trials. Participants received no shocks in the no-shock condition. Same as Shapiro et al. (1984, Experiment 1), except that in the shock condition no shocks were actually presented.

Experiment 2

73% vision-first responses in the noshock condition, 59% vision-first responses in the shock condition 83% vision-first responses in the noshock condition, 64% vision-first responses in the shock condition 67% vision-first responses in the noshock predictable condition, 49% vision-first responses in the no-shock unpredictable condition, 72% vision-first responses in the shock predictable condition, 76% vision-first responses in the shock unpredictable condition In conditions where there were more bimodal than unimodal visual trials, the bimodal RTs were significantly slower than unimodal auditory RTs.

Experiment 3

Same as Shapiro et al. (1984, Experiment 1), except that non-painful tactile stimuli were presented rather than shock stimuli.

Johnson & Shapiro, 1989

Same as Shapiro et al. (1984, Experiment 1), except that two factors (arousal condition; shock or no-shock) and stimulus location predictability (predictable or non-predictable) were varied.

Quinlan, 2000

Same as Colavita (1974, Experiment 4)

Sinnett et al., 2007 Experiments 1-6

In many conditions, there were significantly more visual-only responses than auditory-only responses.

Same as Colavita (1974, Experiment 3), except that on bimodal trials, participants were instructed to press the bimodal response key (i.e., there were three response keys).

27

CHAPTER 1 - INTRODUCTION

Another response requirement that has been used in several previous studies (e.g., Colavita, 1974, Experiment 4; Egeth & Sager, 1977, Experiments 3, 4, & 5; Quinlan, 2000) involved instructing the participants to press the tone key in response to the bimodal targets. Participants tended to make slower tone key responses on the bimodal trials than on the unimodal auditory trials. Egeth and Sager interpreted this slowing of bimodal response latencies (relative to unimodal tone responses) as indicating that the visual component of the bimodal stimulus had ‘interfered’ with the processing of the auditory component of the stimulus (which they defined as an example of visual dominance). However, it could be argued that by requiring their participants to press the tone key in response to the bimodal targets, the participants in Egeth and Sager’s study would presumably have had to suppress any tendency that they may have had to respond to the visual component of the bimodal stimulus. That is, interference in the response selection process may have delayed participants’ tone key responses to bimodal stimuli. It is therefore unclear whether the relative slowing of responses to bimodal targets in Egeth and Sager’s (1977) study of the Colavita effect was really caused by interference in the processing of the bimodal stimuli, or whether instead it may have been caused by interference at the stage of response selection (due to participants having had to suppress their response to the visual component of the bimodal stimuli). Furthermore, any erroneous visual responses made on the bimodal trials would presumably be subject to the same confounds - that is, it would be unclear whether the participants had failed to suppress their visual responses, or whether instead they had simply not perceived the tone at all. Subsequent research has therefore more frequently reverted back to Colavita’s original definition of the Colavita effect (e.g., Johnson & Shapiro, 1989; Shapiro et

28

CHAPTER 1 - INTRODUCTION

al., 1984; Sinnett et al., 2007). It is possible that the type of response requirements used in these early studies of the Colavita effect may have contributed to the large behavioural effect observed (see Sinnett et al., 2007, for a similar point). Therefore, in order to eliminate these potential ambiguities (such as responses biases, response suppression, or experimenter expectancy effects), two different types of response requirements will be explored in the experiments reported in Chapter 2.

1.4.3 Relative stimulus intensities In order to determine whether the relative intensities of the auditory and visual stimuli contributed to the ‘prepotency’ of the visual stimulus over the auditory stimulus in the Colavita visual dominance effect, Colavita (1974) instructed his participants to match the subjective intensity of the two stimuli (in Experiments 1 & 2), or to double the subjective intensity of the tone relative to that of the light (in Experiment 3; see Table 1.2). However, the Colavita effect persisted despite these experimental manipulations. Egeth and Sager (1977, Experiment 6) also explored whether varying the relative stimulus intensities of the auditory and visual stimuli would modulate the extent to which the visual stimulus ‘interfered’ with responses to the auditory stimulus on bimodal trials. They performed a within-participants manipulation in which the auditory and visual stimuli were presented at three ‘light/tone ratios’ (determined subjectively by the participants), where the visual stimulus was either half the subjective intensity of the auditory stimulus, the same subjective intensity as the auditory stimulus, or twice the subjective intensity of the auditory stimulus. Importantly, ‘interference’ was observed in all three conditions, with no significant differences being reported between them.

29

CHAPTER 1 - INTRODUCTION

These findings therefore led both Colavita (1974) and Egeth and Sager (1977) to conclude that the Colavita visual dominance effect was not simply the result of some difference in the relative intensities of the auditory and visual stimuli. Note that while in the early research into the Colavita effect, the auditory and visual intensities tended to be subjectively matched by the participants (e.g., Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979; Egeth & Sager, 1977), this practice has been abandoned by later researchers who have made no specific attempts to match the intensities of the auditory and visual stimuli used (e.g., Johnson & Shapiro, 1989; Quinlan, 2000; Shapiro et al., 1984; Sinnett et al., 2007).
Table 1.2. Table summarizing the relative intensities of the auditory and visual stimuli and the visual dominance effects observed (Colavita, 1974, Experiments 1 & 2; Egeth & Sager, 1977, Experiment 6). At the stimulus intensity ratio of 0.5 the visual stimulus was half the subjective intensity of the auditory stimulus, at the stimulus intensity ratio of 1 the visual stimulus was the same subjective intensity as the auditory stimulus, and at the stimulus intensity ratio of 2 the visual stimulus was twice the subjective intensity of the auditory stimulus.
Study Colavita, 1974 Experiment 1 Experiment 2 Egeth & Sager, 1977 Experiment 6 Stimulus intensity ratio Visual dominance effect

1 0.5

98% visual-only responses 87% visual-only responses

0.5

Bimodal RTs (410ms) significantly slower than unimodal auditory RTs (370ms). Visual dominance observed. Bimodal RTs (432ms) significantly slower than unimodal auditory RTs (382ms). Visual dominance observed. Bimodal RTs (433ms) significantly slower than unimodal auditory RTs (384ms). Visual dominance observed.

1

2

Indeed, it has been shown that while increasing the intensity at which weak (i.e., near-threshold) stimuli are presented will cause participants to perceive them as having been presented earlier in time (e.g., Alpern, 1954; Craig & Baihua, 1990; Efron, 1963; Jaśkowski, 1999; Roufs, 1963; Sanford, 1971; Smith, 1933), such intensity-based effects decline as the stimulus intensity increases (Woodworth & Schlosberg, 1954). Therefore, intensity-based effects may primarily be applicable for stimuli presented at near-threshold levels, whereas interactions occurring between 30

CHAPTER 1 - INTRODUCTION

stimuli which are presented well above threshold might not be affected so much by the variation in the relative intensities of the stimuli (cf. Spence, Shore, & Klein, 2001b). Furthermore, it is even unclear as to what the most appropriate criterion for matching stimuli cross-modally is (i.e., whether they should be matched for simple detection latencies, discrimination latencies, subjective intensities, or according to some other criterion2; see Jaśkowski, 1996; Spence & Driver, 1997c; Spence et al., 2001b; Whipple, Sanford, & Colegrove, 1899, p. 285). For these reasons, no specific attempts were made to match the intensity of the auditory and visual stimuli in the majority of studies reported in this thesis (with the exception of the signal detection study reported in Chapter 8, in which the auditory and visual targets were presented at each participant’s individually-determined 75% detection thresholds).

1.4.4 Arousal Shapiro et al. (1980) demonstrated that pigeons display visual dominance under conditions of ‘normal’ arousal but that they exhibit auditory dominance under conditions of aversive arousal. In order to investigate whether this finding could be extended to humans, Shapiro et al. (1984) presented the participants in their study with the Colavita task (see Table 1.1 for more details), and varied their arousal state. In the aversive arousal state (Experiment 1), the participants were threatened with (and presented with) electric shocks unpredictably throughout the experimental blocks. In the cognitive arousal state (Experiment 2), the participants were threatened with, but not actually presented with, electric shocks. In the non-aversive arousal state (Experiment 3), the participants were informed about and presented with non-aversive

Note that these different means of matching the intensity of the stimuli actually give rise to different results.

2

31

CHAPTER 1 - INTRODUCTION

tactile vibrations unpredictably throughout the experimental blocks. In the control non-aroused state, the participants were not threatened with or presented with electric shocks. Shapiro et al. compared the visual dominance effect reported in the aroused states with the performance observed in the control non-aroused states. They reported that the Colavita visual dominance effect was reversed (i.e., auditory dominance was observed) in the aversive arousal state, and attenuated in the cognitive arousal and non-aversive arousal states. Shapiro et al. suggested that this switch in modality dominance could be due to the fact that audition is more appropriate than vision in aversive situations because of its 360o detection capability.

1.4.5 Spatial confounds The relative spatial locations of the auditory and visual stimuli have not been controlled for in the majority of previous studies of the Colavita effect. In fact, the visual and auditory stimuli have almost always been presented from different spatial locations in previous research (with the exception of the experiments reported recently by Sinnett et al., 2007). For example, Colavita and his colleagues (Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979) always presented the auditory and visual stimuli separated by 15° of visual angle on the horizontal plane (although they counterbalanced which side the stimuli were presented from), with participants being instructed to look directly at the light (or else to focus mid-way between the light and sound sources, or directly at the sound sources, in Colavita & Weisberg’s, 1979, study). Meanwhile, researchers who subsequently followed up on Colavita’s early work presented visual stimuli from directly in front of the participant, with the auditory stimuli being presented over headphones (Egeth & Sager, 1977;

32

CHAPTER 1 - INTRODUCTION

Shapiro et al., 1984), or from different distances and directions with respect to the participant (Johnson & Shapiro, 1989). At present, it is therefore uncertain whether the magnitude of the Colavita effect was affected by the location from which the stimuli were presented. One question here is whether vision may have been found to be dominant in many previous studies of the Colavita effect simply because the participants in these studies were presumably fixating in the direction of the visual stimuli (hence, their attention was also focused there). Therefore, the effect of the relative spatial location (same or different) of the stimuli on the magnitude of the Colavita effect will be explored in Chapter 6. In order to eliminate any possible effects relating to the relative spatial positions of the auditory and visual stimuli, in all of the studies reported in the thesis the auditory and visual stimuli will always appear from the same spatial location (apart from in the experiments in Chapter 6 where the spatial separation between the stimuli was manipulated explicitly). In the next section, the early theories of the Colavita effect will be discussed.

1.5

EARLY THEORIES OF THE COLAVITA EFFECT
Colavita and his colleagues (Colavita, 1974; Colavita et al., 1976) initially

explained the Colavita visual dominance effect in terms of the ‘hard-wiring’ of the auditory and visual systems. They believed that when the stimuli were presented, a ‘reflexive orienting response’ was evoked which involved a change of fixation to the visual stimulus, and argued that this orienting response would be more biased toward the visual stimuli because of vision’s more direct connections with the superior colliculus (cf. Ades, 1944). Using more recent terms, it could be said that Colavita

33

CHAPTER 1 - INTRODUCTION

and his colleagues proposed that the Colavita effect was the result of perceptual processes caused by the visual stimulus being more effective at capturing participants’ (or involuntary) attention exogenously. In contrast to Colavita’s more perceptually-based explanation of the Colavita visual dominance effect, Posner et al. (1976) proposed an attentional account for the Colavita effect (and for many other visual dominance effects). According to this account, the poorer alerting properties of the visual system, as compared to the auditory system (see also Klein, 1977), may have caused the participants in studies of the Colavita effect to bias their attention endogenously toward visual inputs as a means of compensating for this purported weakness (see also Spence, Nicholls, & Driver, 2001a; Whipple et al., 1899). This biasing of attention endogenously toward the visual modality may have resulted in the participants’ failure to respond to the auditory stimuli on some proportion of the bimodal trials. Support for this argument has also come from subsequent research reported by Egeth and Sager (1977). They reported that endogenous manipulations of attention elicited either by changing the stimulus probabilities (increasing the frequency of bimodal stimuli relative to unimodal visual stimuli; also see Quinlan, 2000) or by simply instructing their participants to ‘attend to their ears’, decreased the extent to which visual dominance occurred (in terms of the visual stimulus ‘interfering’ with the responses to bimodal stimuli; see Chapter 4, for a more detailed description of these manipulations). These findings led many researchers, including Colavita himself, to conclude that the Colavita visual dominance effect is caused by participants directing their attention endogenously toward the visual modality under normal conditions (Colavita & Weisberg, 1979; Egeth & Sager, 1977; Klein, 1977; Posner et al., 1976; Quinlan, 2000; Shapiro et al., 1984). However, the argument that

34

CHAPTER 1 - INTRODUCTION

response biases or response conflicts may have contributed to the ‘interference’ observed in Egeth and Sager’s (1977) study (as discussed in Section 1.4.2) makes the evidence provided by Egeth and Sager (1977), at best, equivocal. Hence, as there is presently no clear answer to the question of whether or not the modality toward which participants direct their attention (either endogenously or exogenously) does contribute to the Colavita effect, these questions will be explored in depth in Chapter 4. Recent research into the nature of audiovisual interactions suggests that a number of factors apart from attention, such as the spatial separation, temporal separation, and semantic congruency, between the auditory and visual stimuli may also contribute to the Colavita effect. The potential contribution of these factors to the Colavita effect will be explored in the following section.

1.6 MULTISENSORY INTEGRATION AND THE UNITY EFFECT
As we interact with the rich and varied multisensory environment around us, our brains constantly integrate the information striking the different sensory receptors in order to generate the unified multisensory perceptual experiences that fill our daily lives (Driver & Spence, 2000). The purpose of combining these inputs is presumably to enhance our ability to discriminate the occurrence of one or more of the events taking place in the environment, and to react to these events effectively (Calvert et al., 2004). The question arises, however, as to how we determine which sensory inputs can be attributed to the same distal event and which sensory inputs belong to distinct environmental events.

35

CHAPTER 1 - INTRODUCTION

According to research on the ‘unity effect’ (e.g., Bedford, 2001; Spence, 2007; Vatakis & Spence, 2007b; Welch & Warren, 1980), the stronger an observer’s assumption that two events refer to the same object (rather than to two separate events), the greater the multisensory interaction (or intersensory bias) between them. An extensive body of research has shown that the extent to which multisensory signals are integrated is determined by the structural properties of the audiovisual inputs, such as their spatial separation and temporal synchrony (see Calvert et al., 2004; Vatakis & Spence, 2007b for reviews). Indeed, many visual dominance effects are modulated by the degree of spatial separation between the stimuli (e.g., for dynamic stimuli: Soto-Faraco et al., 2002), as well as the degree of temporal separation between them (e.g., for the ventriloquist effect: Choe, Welch, Gilford, & Juola, 1975; Radeau & Bertelson, 1987; Thomas, 1941; for dynamic stimuli: SotoFaraco et al., 2002; see Soto-Faraco & Kingstone, 2004, for a review; for the McGurk effect: Dixon & Spitz, 1980; Munhall et al., 1996). Another factor that has been shown to modulate audiovisual integration is the semantic congruency between the auditory and visual stimuli (i.e., whether or not the stimuli refer to the same type of object or event; Laurienti et al., 2004; Molholm et al., 2004; Taylor et al., 2006; Vatakis & Spence, 2007b). Thus, as many audiovisual interactions, and especially visual dominance effects, appear to be modulated by factors that contribute to the unity effect, it is plausible to suggest that these factors may also modulate the Colavita effect. The effects of spatial and temporal separation, and semantic congruency will therefore be explored in Chapters 5, 6, and 7, respectively.

36

CHAPTER 1 - INTRODUCTION

1.7 THE COLAVITA EFFECT AND CROSSMODAL EXTINCTION
One of the reasons why the study of the Colavita effect in normal participants may be of particular interest is in terms of its putative similarity to the clinical phenomenon of extinction observed in neuropsychological patients (e.g., Bender, 1952; Bueti, Costantini, Forster, & Aglioti, 2004; Costantini, Bueti, Pazzaglia, & Aglioti, 2007; di Pellegrino, Làdavas, & Farné, 1997b; Frassinetti, Pavani, & Làdavas, 2002; Mattingley, Driver, Beschin, & Robertson, 1997; Rapp & Hendel, 2003; Sarri, Blankenburg, & Driver, 2006). Extinction is a neurological condition typically caused by a unilateral stroke, in which patients are able to detect contralesional events when they are presented in isolation, but tend to miss them when they are presented together with another stimulus on the ipsilesional side (see Maravita, Husain, Clarke, & Driver, 2001, for a description of crossmodal extinction). In the case of extinction, it is the ipsilesional stimulus that reliably extinguishes the patient’s awareness of the contralesional stimulus on a proportion of the trials, whereas in the Colavita effect it is the presentation of the visual stimulus that appears to extinguish a participant’s awareness of the auditory stimulus on a certain proportion of trials. On account of the apparent similarity between extinction and the Colavita effect, this thesis therefore puts forward the argument that Colavita effect can therefore perhaps be thought of as a non-pathological form of crossmodal extinction. Indeed, forms of within-modality extinction have been previously observed in neurologically normal participants (see Gorea & Sagi, 2002, and Bender, Green, & Fink, 1954, for examples of extinction in normal participants within the visual, or

37

CHAPTER 1 - INTRODUCTION

tactile modalities, respectively3), however, the notion of crossmodal extinction in normal participants has rarely (if ever) been discussed in the literature. The similarity between the Colavita effect that occurs in normal participants and the phenomenon of crossmodal extinction observed in neuropsychological patients may make the insights gained in study of the Colavita effect particularly relevant to understanding the phenomenon of extinction, and vice versa. One noteworthy advantage of studying the Colavita effect, in order to gain insights into extinction, is that the Colavita effect can be shown in normal populations and is therefore easier to investigate. In contrast, patients displaying extinction are rare and often have a variety of deficits that change over time. Similarly, the study of the Colavita effect can benefit from, and gain inspiration from, the theoretical developments and ideas from the extinction literature (e.g., see Chapters 5 and 8).

1.8 FACTORS THAT MAY MODULATE THE COLAVITA EFFECT
To summarise, the Colavita effect is a remarkable audiovisual interaction that has been reported in the literature on visual dominance. It provides a striking example of how a visual stimulus can dominate our perception so much that it apparently precludes the concurrent perception of (or at least response to) an auditory stimulus. The Colavita effect bears an interesting parallel to the phenomenon of extinction that has been observed in neuropsychological patients, and it would seem possible that the study of the Colavita effect in normal participants may eventually hold the promise of

Gorea and Sagi (2002) reported that when participants are presented with two visual stimuli of varying visibility, the less visible targets are extinguished. Bender et al. (1954) reported that for participants presented with two simultaneous tactile stimuli on different parts of their body, the tactile stimuli presented to the more dominant part of the body (e.g., face or erogenous zones) tended to extinguish tactile stimuli presented elsewhere.

3

38

CHAPTER 1 - INTRODUCTION

providing a model of crossmodal extinction in patients. The early studies of the Colavita effect were rife with confounds, and even 33 years after the Colavita effect was first reported, this potentially important phenomenon remains an enigma. Therefore, the aim of the experiments reported in this thesis will be to identify the factors that modulate the Colavita visual dominance effect and to thus gain a better insight into the mechanisms behind the phenomenon. In order to find the optimal methodology with which to study the Colavita visual dominance effect, the effects of different response requirements (Chapter 2) and stimulus probabilities (of the unimodal auditory, unimodal visual, and bimodal stimuli; Chapter 3) will be explored in the first of the experimental chapters. Following this, the effects of directing a participant’s attention toward the auditory or visual modality, either exogenously or endogenously, on the Colavita effect will be investigated (Chapter 4). The effects of spatial separation, temporal separation, and semantic congruency on the magnitude of the Colavita effect will be tested in Chapters 5, 6, and 7, respectively. Finally, the differential contributions of perceptual and post-perceptual processes to the Colavita effect will be investigated in Chapter 8.

39

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

CHAPTER 2
2.0 RESPONSE REQUIREMENTS AND RESPONSE BIASES
In Colavita’s (1974) seminal study, the participants were instructed to press an auditory response key (the ‘tone key’) whenever they heard a tone and a visual response key (the ‘light key’) as soon as they saw a light. However, the participants were not informed that bimodal stimuli might be presented, nor were they instructed as to how to respond on such trials. In order to remove any confounds arising from this ‘deceit’, subsequent researchers have typically informed their participants that bimodal trials would be presented and have explicitly instructed them as to how to respond on those trials. Over the years, a variety of different response requirements have been used to investigate the Colavita effect. However, due to certain factors (e.g., responses biases, response suppression, and/or experimenter expectancy effects; see Chapter 1, Section 1.4.2), it is possible that the types of response requirements used in those early studies may have contributed to the large behavioural visual dominance effects that were observed (cf. Sinnett et al., 2007, on this point). In order to eliminate these potential ambiguities attributable to the specific instructions and response requirements used in the previous studies of the Colavita effect, the participants in Experiment 2.1 were explicitly instructed to press both the auditory and visual response keys whenever a bimodal target was presented, the 40

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

auditory response key for a unimodal auditory stimulus (a tone), and the visual response key for a unimodal visual stimulus (a light) (hereafter, this response requirement will be termed the two-key response condition). This response requirement has the advantage that it avoids the necessity of participants having to suppress their responses to either the visual or auditory components of the bimodal stimulus. A number of researchers have used this type of response requirement previously (Colavita & Weisberg, 1979; Egeth & Sager, 1977; Shapiro et al., 1984). However, it should be noted that they defined the Colavita visual dominance effect as occurring when participants responded to the visual stimulus first on bimodal trials (hence, it is possible that any response bias that participants may have had to press the visual response key first may have contributed to the visual dominance obtained in those studies). In contrast, in all of the experiments reported in this thesis, the Colavita effect is defined as occurring if the percentage of bimodal trials in which the participants made visual-only responses (i.e., when the participant only pressed the visual response key) is greater than the percentage of auditory-only responses, and its magnitude is calculated as the difference between the percentage of visual-only and auditory-only responses on the bimodal trials. The primary aim of the two experiments reported in this chapter was therefore to investigate whether the Colavita visual dominance effect could be replicated, using stimuli similar to those used in Colavita’s original studies, while at the same time ruling out the response selection confounds present in the majority of the earlier studies of the phenomenon. Note, however, that one might expect the overall prevalence of visual-only responses to be somewhat lower than that reported in Colavita’s original studies (cf. Chapter 1, Table 1.1), given the elimination of certain factors (such as the use of deception - Colavita’s failure to give any clear instructions

41

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

as to how the participants should respond on bimodal trials, and any effects arising from the particular response requirements of the task) that may have contributed to the Colavita effect reported in the previous research. In Experiment 2.1, the Colavita effect was investigated using a speeded discrimination task (where the participants had to discriminate whether the stimulus was a unimodal auditory, unimodal visual, or bimodal stimulus as quickly as possible)6 and the two-key response condition.

2.1

EXPERIMENT 2.1

2.1.1 Methods Participants. 14 naïve participants (mean age of 24 years, age range from 1833 years; 7 males and 7 females) took part in Experiment 2.1. All except three of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes. In this, and all subsequent studies reported in this thesis, all of the participants were given one £5 gift voucher in return for taking part in the study (with the exception of studies that lasted for longer than 45 minutes in which case they were given two vouchers), and had normal or corrected-to-normal sight and normal hearing. Apparatus and materials. The participants sat approximately 60cm from the light and sound sources in a dimly-illuminated testing booth. The visual stimulus consisted of the illumination of a yellow light emitting diode (LED) with a luminance of 1.9cd/m2 placed directly in front of the participant at eye-level for 50ms. The auditory stimulus consisted of a 4000Hz pure tone presented for 50ms from a

6

It should be noted that the task that participants had to perform could also be considered to be a modality detection task. Quite what the most appropriate description for the task is, is somewhat ambiguous. The participants were both detecting stimuli in different sensory modalities, and discriminating which modality a particular target was presented in.

42

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

loudspeaker cone situated 2cm directly behind the LED, such that the auditory and visual stimuli appeared to emanate from exactly the same spatial position. The tones were presented at 65dB(A), as measured from the participant’s ear position. Note that both the light and the tone were presented at an intensity that was well above threshold levels and were therefore clearly perceived by the participants. Amplitude enveloping was applied to the first and last 5ms of the tone stimuli, using the Adobe Audition 1.5 audio editing software. Responses were collected from a computer keyboard placed on a table directly in front of the participant.
60cm Light and sound source

Figure 2.1. Schematic diagram of the experimental set-up of Experiment 2.1

The participants were instructed to press one key in response to the auditory stimuli (the auditory response key) and another key in response to the visual stimuli (the visual response key), with the allocation of the stimuli to the response keys (the ‘n’ and ‘m’ keys) counterbalanced across participants. Participants used the index and middle fingers of their right hand to respond. The participants were instructed to press both response keys whenever a bimodal stimulus was presented (i.e., when an auditory and visual stimulus were presented at the same time). Note that no specific instructions were provided to the participants as to whether they should press the two response keys simultaneously or not. The experiment was controlled using the EPrime software (Schneider, Eschman, & Zuccolotto, 2002a, b).

43

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

Design. The participants were presented with 6 blocks of 100 trials. There were 40 visual, 40 auditory, and 20 bimodal trials in each block (the same stimulus probabilities have also been used in a number of previous studies, e.g., Egeth & Sager, 1977; Quinlan, 2000; Sinnett et al., 2007; and are similar to those used in Colavita’s original research, e.g., Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979). The order of stimulus presentation was randomised within each block of trials. A block of 30 practice trials was presented before the main experimental session. The practice trials were identical to the main experimental trials but were not analysed. Procedure. On each trial, the participants were presented with an auditory, visual, or bimodal target. The target was presented for 50ms at the start of each trial, followed by a 1750ms response interval (i.e., successive targets were always separated by 1800ms7). After the end of a trial, the next trial began automatically. It is important to note that the participants had enough time in which to make a second response, should they have wanted to, after having made their initial response (provided, that is, that they did so within the response interval before the onset of the target on the next trial). The participants were instructed to respond as rapidly and accurately as possible. No feedback regarding the correctness of the participant’s responses was provided.

7

Note that although the interstimulus interval (ISI) used in Experiment 2.1 was much shorter than that typically used in previous studies of the Colavita effect (1750ms, as compared to 15,000ms, 7000ms, and 3000ms in Colavita, 1974; Colavita & Weisberg, 1979; Egeth & Sager, 1977, respectively), the participants nevertheless had sufficient time in which to respond. Indeed, the participants tended to respond well before the end of the 1800ms response interval (see the mean RTs in Table 2.1).

44

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

2.1.2 Results
Table 2.1. Mean error rates for the unimodal auditory, unimodal visual, and bimodal target stimuli in Experiment 2.1. Note that there were two types of error on the bimodal trials: Participants could either make an auditory-only or a visualonly response. There were also two types of errors participants would make on unimodal trials: Bimodal responses and inappropriate responses. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. Bimodal auditory and visual responses refer to the RTs to the auditory and visual components of the bimodal target stimuli (i.e., auditory or visual keypresses). Standard errors are shown in parentheses.

Error rates (%) Unimodal auditory Bimodal responses Inappropriate responses Unimodal visual Bimodal responses Inappropriate responses

3.9 (0.9) 2.3 (0.6) 1.7 (0.5) 4.1 (0.7) 2.5 (0.5) 1.6 (0.4)

Bimodal Auditory-only responses Visual-only responses

4.4 (1.8) 12.1 (2.5)

RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses

636 (32) 577 (27)

715 (32) 685 (30)

Participants failed to respond on 2.1% of the trials, and these trials were not included in the data analyses. The results are shown in Table 2.1. There were two types of errors that participants could make to bimodal target stimuli: Visual-only responses (i.e., pressing the visual but not the auditory response key), and auditoryonly responses (i.e., pressing the auditory but not the visual response key). The percentage of visual-only (or auditory-only) responses represents the percentage of the total number of bimodal trials presented (e.g., 120 trials per participant in Experiment 2.1) in which participants made a visual-only (or auditory-only) error. 45

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

There were two types of error that participants could make on unimodal trials (remember, null responses were removed from the analyses): Bimodal responses (i.e., pressing both responses keys, rather than just the relevant response key on its own), and inappropriate responses (i.e., pressing the irrelevant response key and not pressing the relevant response key; e.g., on unimodal auditory trials, this would involve pressing the visual response key but not the auditory response key). Pooling of the unimodal error data. When analysing the unimodal error data, the incorrect bimodal responses and inappropriate response data were pooled together for the analyses of the unimodal error data for the following two reasons. First, the type of error that participants made on unimodal trials was not a variable of interest (the main point of interest was whether there was a Colavita effect, and what its magnitude was), thus pooling the data would simplify the data analysis. Secondly, both the bimodal responses and the inappropriate responses give an indication of the percentage of unimodal trials in which participants made a response to the irrelevant stimulus (where participants either also responded to the relevant stimulus on bimodal response trials, or did not respond to the relevant stimulus on inappropriate response trials). Analysis of the unimodal error data suggests that the main difference between the different types of unimodal error responses lies in the decision by participants as to whether or not to respond to the relevant stimulus (given that they had already responded incorrectly by responding to the irrelevant stimulus), rather than differences in what the participants may actually have perceived. An analysis of the incorrect bimodal response data revealed that participants made significantly more rapid visual than auditory responses on unimodal auditory trials (mean difference = 73ms; t(13) = 2.46, p = .028), and significantly more rapid auditory than visual responses on unimodal visual trials (mean difference = 117ms;

46

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

t(13) = 3.07, p = .009), suggesting that when participants made bimodal responses, they first made a false alarm to the irrelevant stimulus and then subsequently made a response to the relevant stimulus. The fact that participants tended to respond faster to the irrelevant stimulus than to the relevant stimulus suggests that the error was caused by participants incorrectly anticipating the irrelevant stimulus. It could have been the case that when participants made inappropriate responses, they first incorrectly responded to the irrelevant stimulus (as they appeared to do when they make bimodal responses) but then did not subsequently respond to the relevant stimulus (e.g., because they realised that their response was incorrect already, so there was no point in making the response to the relevant stimulus). The finding that there were no significant differences between the percentage of incorrect bimodal responses and inappropriate responses for both unimodal auditory and for unimodal visual stimuli (t(13) = 0.86, p = .407; t(13) = 1.35, p = .201) shows that there was no consistent pattern between participants as to whether they tended to make bimodal responses or inappropriate responses: some participants tended to make bimodal responses, others tended to make inappropriate responses. This therefore suggests that there may have been a lot of variation between the participants as to whether or not they did decide to make a response to the relevant stimulus (given they that had already responded to the irrelevant stimulus). To summarise, the unimodal error were pooled together, firstly, because the different kinds of errors participants made on unimodal trials was not a variable of particular interest to the investigation of the Colavita effect. Second, the analyses conducted on the unimodal error data suggest that the main difference between the different types of unimodal error responses is due to a decision that participants made (a decision that appeared to vary between participants) as to whether or not to respond

47

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

to the relevant stimulus given that they had already made a false alarm to the irrelevant stimulus. Thus, as both the bimodal responses and the inappropriate responses are indicative of trials in which participants made a response to the inappropriate stimulus, the data were pooled together. Bimodal target data. The data from the bimodal error trials (i.e., the bimodal trials in which the participants failed to respond correctly) were analysed using an analysis of variance (ANOVA) with the factor of Response (Auditory-only or Visualonly). In this and all subsequent analyses reported in this thesis, Greenhouse-Geisser corrections were made whenever the assumption of sphericity was violated. The analysis revealed a significant main effect [F(1, 13) = 6.45, p = .025]. Participants made significantly more visual-only than auditory-only responses (12.1% vs. 4.4% of all bimodal trials, respectively); that is, the participants failed to respond to the auditory component of the bimodal targets more often than they failed to respond to the visual component, thus demonstrating a robust Colavita effect. A similar ANOVA performed on the RT data from the bimodal error trials revealed no significant main effect of Response [F < 1, n.s.]. Note that a certain number of the participants always responded to the auditory and/or visual component(s) of the bimodal targets (3 participants), or else did not provide at least 10 visual-only and 10 auditory-only RTs. The data from these participants were therefore not included in this analysis of the RT data, as they did not provide a sufficient number of responses with which to arrive at a meaningful estimate of their mean RT. Due to the fact that participants were tested in a number of experimental conditions in many of the subsequent studies in this thesis, this meant that in most cases the bimodal RT data could not be analysed (because there were too few trials

48

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

per condition). Therefore, the bimodal RT data were not analysed in any of the subsequent experiments in this thesis. Error data. An ANOVA performed on the error data8 revealed a significant main effect of Target Stimulus (Auditory, Bimodal, or Visual) [F(1.03, 13.39) = 21.27, p < .001], due to participants responding less accurately on the bimodal target trials (16.5% errors) than on either the unimodal auditory (3.9% errors; t(13) = 5.15, p < .001) or the unimodal visual target trials (4.1% errors; t(13) = 4.29, p = .001), but no less accurately on unimodal visual than on unimodal auditory target trials (t(13) = .32, p = .758). RT data. The RT data from those trials in which the participants responded correctly were initially analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), and Target Type (Unimodal or Bimodal). The analysis revealed a significant main effect of Target Type [F(1, 13) = 31.15, p < .001], attributable to the participants responding significantly more rapidly on the unimodal (607ms) than on bimodal target trials (700ms)9. There was also a significant main effect of Target Modality [F(1, 13) = 41.52, p < .001], with the participants responding more rapidly to visual (631ms) than to auditory targets (675ms), overall. Finally, the interaction between Target Modality and Target Type was also significant [F(1, 13) = 7.31, p = .018], due to the difference between the visual and auditory

The analysis was re-run on the arcsine-transformed error data to ensure normality and homogeneity of variance, and produced the same results as the original analysis. There was a significant main effect of Target Stimulus (Auditory, Bimodal, or Visual) [F(1.09, 14.20) = 21.16, p < .001], due to participants responding less accurately on bimodal target trials (P = 0.39; where P is the arcsine-transformed proportion) than on either unimodal auditory (P = 0.19; t(13) = 5.60, p < .001) or unimodal visual target trials (P = 0.19; t(13) = 4.14, p = .001), but no less accurately on unimodal visual than on unimodal auditory target trials (t(13) = .67, p = .515). 9 It is interesting to note that participants in this experiment, and many of the subsequent experiments in this thesis, responded more rapidly to unimodal than to bimodal targets, which is the reverse pattern of response latencies observed in redundant target paradigms (Miller, 1982). The main difference between the Colavita effect and redundant target paradigm lies in the response requirements of the tasks: in the former, participants make judgements in which they have to discriminate the modality of the targets, whereas in the latter they have to make simple detection responses instead.

8

49

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

response latencies being more pronounced for the unimodal targets than for the bimodal targets (mean differences = 59ms vs. 30ms, respectively; t(13) = 2.67, p = .019).

2.1.3 Discussion The results of Experiment 2.1 revealed a clear Colavita effect; that is, when the participants failed to respond correctly on the bimodal target trials (which they did on 16.5% of all bimodal trials), they made significantly more visual-only than auditory-only responses (12.1% vs. 4.4% of all bimodal trials, respectively). Note, though, that the magnitude of the Colavita effect reported in Experiment 2.1 (the percentage of visual-only responses minus the percentage of auditory-only responses; 7.7%) is much smaller than that reported in Colavita’s (1974) original study (96%; see Chapter 1, Table 1.1). This difference most likely reflects the lack of response biases and deception in Experiment 2.1 that may have contributed to the Colavita effect observed in the original study. The fact that the Colavita effect emerged even when using these unambiguous response requirements suggests that the response requirements of the task are not the main factor contributing to the Colavita effect (at least for the effect reported here). Analysis of the RT data from the correct response trials revealed that the participants responded significantly more rapidly to visual than to auditory stimuli on both the unimodal and bimodal target trials. It could therefore be argued that the occurrence of the Colavita effect in Experiment 2.1 may have been due to participants responding more rapidly to the visual component than to the auditory component of the bimodal target stimuli. The law of prior entry (originally formalised by Titchener in 1908) states that attended stimuli are perceived more rapidly than the same stimuli 50

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

when they are not attended (see Spence et al., 2001b, for a review). In other words, if prior entry is responsible for the Colavita effect (either because visual stimuli exogenously capture attention, or because participants, for whatever reason, endogenously attend to the visual modality; for a discussion, see Chapter 4), then participants will tend to perceive stimuli presented in the visual modality as having been presented earlier in time than those in the auditory modality, thus potentially giving rise to faster responses to visual stimuli10. Hence, it could be argued that the more rapid RTs observed in response to visual stimuli may be a product, but not the cause, of the Colavita visual dominance effect. In addition, a Pearson correlation analysis performed (on an across-participant basis) on the bimodal error data and the RT data for the bimodal targets revealed no significant correlation (r(14) = +0.012. p = .967) between participants’ relative speed of responding to auditory and visual bimodal targets (i.e., the extent to which their auditory RTs were delayed relative to their visual RTs) and the magnitude of the Colavita effect that they exhibited. This supports the argument that participants’ relative speed of responses to the auditory and visual targets does not modulate the magnitude of the Colavita effect. Furthermore, the Colavita effect has been reported to occur even when the auditory RTs were faster than the visual RTs (Sinnett et al., 2007), or when there was no significant difference between the RTs to auditory and visual stimuli (Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979). Indeed, it will be demonstrated in Experiment 2.2 that the Colavita effect still occurs when participants respond more rapidly to unimodal auditory targets than to unimodal visual targets. Hence, there is evidence and arguments to support the notion that the

10

Indeed, the experiments reported in Chapter 4 will demonstrate that endogenous and exogenous attention do contribute to the magnitude of the Colavita effect.

51

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

Colavita effect is not simply caused by participants’ relative speed of responses to the auditory and visual components of bimodal target stimuli.

2.1.4 Sequence effects Anecdotal reports from participants suggested that they made more mistakes after being presented with certain combinations of stimuli. Hence, in order to investigate the effect that the preceding stimulus had on the type of errors participants made on bimodal trials, an ANOVA with the factors of Response (Auditory or Visual), and Preceding Stimulus (Auditory, Visual, or Bimodal) was performed on the bimodal error data. There was a significant main effect of Response [F(1, 13) = 6.91, p = .021], with participants making more visual (12.1% of all bimodal trials) than auditory responses (4.4% of all bimodal trials); that is, there was a significant Colavita visual dominance effect. The analysis also revealed a significant main effect of Preceding Stimulus [F(2, 26) = 13.60, p < .001], attributable to participants making fewer errors on bimodal trials when the preceding stimulus was a bimodal stimulus (2.1% errors on bimodal trials which had been preceded by a bimodal stimulus) than when it was a unimodal auditory (8.7% errors on bimodal trials which had been preceded by an auditory stimulus; t(13) = 4.15, p = .001) or a unimodal visual stimulus (5.7% errors; t(13) = 4.7, p < .001), and more errors when the preceding stimulus was unimodal auditory than unimodal visual (t(13) = 2.41, p = .0.32). The interaction between Response and Preceding Stimulus failed to reach statistical significance [F(2, 26) = 1.97, p = .160]. In sum, the analysis revealed that the preceding stimulus had no significant effect on the type of error that participants made on bimodal trials, supporting the argument that the Colavita effect is not an artefact of a response bias participants may 52

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

have to either to persevere with the same response as they made on the previous trial, or to press a different key than the one they pressed on the previous trial. When responding to bimodal trials, participants responded more accurately when the preceding stimulus was a bimodal stimulus than when it was a unimodal auditory or a unimodal visual stimulus, but no more accurately when the preceding stimulus was a unimodal visual than a unimodal auditory stimulus. According to the literature on repetition effects, when the same stimulus is repeated in speeded-choice tasks, responses are faster and more accurate than on non-repetition trials (this phenomenon is known as the repetition effect; Bertleson, 1961; Pashler & Baylis, 1991; Soetens, Melis, & Notebaert, 2002; repetition effects are discussed in greater detail in Chapter 3). This could provide a possible explanation for why participants responded more accurately and rapidly to unimodal stimuli than to bimodal stimuli; the stimuli that occurred with a higher probability (i.e., the unimodal auditory and unimodal visual trials, which both occurred on 40% of the trials) would be more likely than infrequent stimuli (i.e., bimodal stimuli, which occurred on only 20% of the trials) to have samestimulus repetitions. Thus, unimodal stimuli would be more likely to benefit (i.e., in terms of lower error rates and faster RTs) from stimulus repetition benefits than would bimodal stimuli. Indeed, the experiments reported in Chapter 3 will investigate what happens when the relative probabilities of the three types of target are manipulated, and when the bimodal stimulus is no longer presented on the minority of trials.

2.1.5 Response coupling Fagot and Pashler (1992) reported a study of dual-task performance in which their participants had to make two different responses to the same aspect of a single 53

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

target stimulus. After some practice, Fagot and Pashler reported that their participants were able do this with only a single act of response selection; that is, both responses appeared to have been selected and initiated at the same time (cf. Holender, 1980, for a similar model of dual-task response initiation). This phenomenon is known as response coupling. By contrast, when the participants in Fagot and Pashler’s study had to make two responses to different attributes of the same object (or else to two different objects), response selection occurred in a serial manner; that is, there appeared to be a time-locked delay in the selection and initiation of the second response relative to that of the first (i.e., a psychological refractory period, PRP, effect was demonstrated; see Pasher, 1998, for a review). On the unimodal trials in Experiment 2.1, the participants responded to visual stimuli an average of 59ms more rapidly than they responded to the auditory stimuli. If separate response selection and initiation processes were required for the two components of the bimodal targets, then one would have expected the difference in response latencies (between the responses to the auditory and visual components of the bimodal targets) to have been of a similar magnitude as for the unimodal target stimuli. However, this response latency difference for bimodal targets (30ms) was significantly smaller than that observed for the unimodal target stimuli (59ms). Furthermore, analysis of the data from the bimodal trials in which the participants correctly made both responses revealed that on 75% of the bimodal trials the visual and auditory response latencies actually fell within 30ms of each other, while on a third of those trials (i.e., 25% of the bimodal trials), this difference was smaller than 10ms. On the remaining 25% of the bimodal trials in which the response latency difference exceeded 30ms, the average latency difference was actually greater than 200ms (mean = 207ms), with participants responding more rapidly to the visual

54

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

component of the bimodal stimulus than to the auditory component on 87% of these trials. Thus, it appears that the difference between the auditory and visual RTs on bimodal trials was driven by those trials in which participants responded first to the visual stimulus, and responded only later to the auditory stimulus when they realised that it had been presented. In order to determine whether there was any evidence of response coupling in Experiment 2.1, it was necessary to test whether there was any correlation between the response latencies to the visual and auditory components of the bimodal target stimuli. Two analyses were performed on the data. First, a Pearson correlation was performed on the means of the auditory and visual RTs across participants (on bimodal trials in which participants responded correctly, by pressing both the auditory and visual response keys), which revealed a significant correlation (r(12) = 0.997, p < .001; see Figure 2.2A), thus suggesting that, overall, participants’ mean auditory RTs were correlated with their mean visual RTs. However, in order to investigate whether there was a correlation between the auditory and visual RTs on a trial-by-trial basis within the individual participants (i.e., whether the participants’ auditory RTs were correlated with their visual RTs once differences between the participants had been taken into account), a second analysis, an analysis of covariance (ANCOVA), was performed on the data (see Bland & Altman, 1995, for a explanation of this statistical technique). The ANCOVA is a combination of regression and ANOVA statistical procedures; the regression removes the effects of the sample size, while the ANOVA provides a sensitive test of the adjusted data (Cochran, 1957; Fisher, 1932). Across the 14 participants, there were 1647 correct responses on the bimodal trials (see Figure 2.2B). The ANCOVA revealed a significant correlation [F(1, 1632) = 2167.5, p < .001], thus supporting the view that on the majority of the bimodal trials

55

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

in which participants responded correctly, the participants’ responses to the auditory and visual components of the bimodal stimulus were coupled (that is, they formed a response couplet). Hence, presumably a single act of response selection occurred as participants initiated both responses together. It would therefore appear that the participants in Experiment 2.1 tended to respond to the bimodal target stimuli as if they constituted individual audiovisual targets, and may have been responding to the single target attribute of bimodality.

A
2000 1500

Visual RTs

1000

500

0 0 500 1000 Auditory RTs 1500 2000

B
2000 1500

Visual RTs

1000

500

0 0 500 1000 Auditory RTs 1500 2000

Figure 2.2. Scatterplots highlighting the RTs to the visual and auditory components of the bimodal targets, for those trials in which participants made a correct response, in Experiment 2.1. In Figure 2.2A, each point represents the average auditory RT and the average visual RT for a single participant, whereas in Figure 2.2B, each dot represents an individual trial across all of the participants.

56

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

On the trials in which the participants’ responses did not appear to be coupled (i.e., on the 25% of bimodal trials in which the average latency difference was 207ms), the latency difference may have been caused by participants responding to the visual component of the bimodal target stimulus as if it were, in fact, a unimodal visual target, and only responding to the auditory stimulus if and when they later ‘noticed’ that it had been presented. To support this argument, the average RT for visual responses on those trials where the response latency difference was greater than 30ms was 585ms (which does not differ significantly from the average mean RT reported for correct responses to unimodal visual stimuli; i.e., 577ms). In contrast, the average RT for auditory responses on those trials where the response latency difference was greater than 30ms was 792ms (which does differ significantly from the average mean RT reported for correct responses to unimodal auditory stimuli; i.e., 636ms).

2.1.6 Response selection and the Colavita effect It is possible that participants may have responded less rapidly and accurately to bimodal stimuli due to a cost associated with their having to make two responses to a bimodal target (i.e., having to press two response keys) as compared to making just a single response (i.e., having to press one response key) in response to the unimodal targets. In other words, the response requirements used in Experiment 2.1 may have led to the difficulty (in terms of both the speed and accuracy of responding) which participants seemingly experienced when responding to the bimodal targets - despite the fact that their responses to bimodal targets appear to have been coupled. In order to examine the extent to which the specific response requirements of the task contributed to the Colavita effect, the participants in Experiment 2.2 were 57

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

presented with somewhat different response requirements. The participants were given three response keys (hereafter, the three-key response condition; cf. Sinnett et al., 2007, who also used the same response requirements). The participants were instructed to press the auditory response key for unimodal auditory targets, the visual response key for unimodal visual targets, and a third bimodal response key for bimodal targets. Using such a design, the response requirements for the unimodal and bimodal targets were equalised (i.e., only a single keypress was required in each case).

2.2

EXPERIMENT 2.2

2.2.1 Methods Participants. 12 new naïve participants (mean age of 25 years, age range from 19-30 years; 7 males and 5 females) took part in Experiment 2.2. All of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes. Apparatus, materials, design, and procedure. The apparatus, materials, and design were exactly the same as in Experiment 2.1 with the exception that the participants now had three response keys, one for auditory, one for visual, and one for bimodal targets. The allocation of the stimuli to the response keys (the ‘b’, ‘n’, and ‘m’ keys) was counterbalanced across participants. Participants used the index, middle, and ring fingers of their right hand to respond. On each trial, a 1000ms interval in which nothing was presented preceded the presentation of the target. Following the presentation of the target, the participants had a 1950ms interval in which to respond. The next trial began as soon as the participant had responded. If no response had been made by the end of the response interval, the 58

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

next trial started automatically. Hence, the interstimulus interval (ISI) in Experiment 2.2 (a 1000ms interval together with a mean RT of approximately 600ms) was similar to that used in Experiment 2.1 (i.e., 1750ms).

2.2.2 Results Error data. The participants failed to respond on 0.2% of the trials overall, and these trials were not included in the data analyses. The data from Experiment 2.2 are shown in Table 2.2. The bimodal trials in which the participants failed to respond correctly were analysed in an ANOVA with the factor of Response (Auditory-only or Visual-only). This analysis showed that the participants made significantly more visual-only responses (9.6% of all bimodal trials) than auditory only responses (3.2% of all bimodal trials) [F(1, 11) = 33.95, p < .001], demonstrating a significant Colavita effect.
Table 2.2. Mean error rates for the unimodal auditory, unimodal visual, and bimodal target stimuli in Experiment 2.2. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. Standard errors are shown in parentheses.

Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses

2.1 (0.4) 3.4 (0.8)

3.2 (0.6) 9.6 (0.8)

RTs (ms) Unimodal auditory Unimodal visual Bimodal

539 (28) 582 (28) 641 (30)

59

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

An ANOVA performed on the error data, with the factor of Target Stimulus (Auditory, Bimodal, or Visual), revealed a significant main effect [F(2, 22) = 95.00, p < .001]. Participants made significantly more errors on the bimodal target trials (12.8% errors) than on either the unimodal auditory (2.1% errors; t(11) = 13.80, p < .001), or unimodal visual trials (3.4% errors; t(11) = 8.71, p < .001). The participants also made more errors on unimodal visual than on unimodal auditory trials (t(11) = 2.10, p = .056), although this latter trend just failed to reach statistical significance. RT data. A similar ANOVA conducted on the RT data, from those trials in which the participants responded correctly, also revealed a significant main effect of Target Stimulus [F(2, 22) = 23.51, p < .001]. Participants responded significantly more rapidly to unimodal auditory targets (539ms) than to either unimodal visual (582ms; t(11) = 5.38, p < .001) or to bimodal targets (641ms; t(11) = 6.24, p < .001), and more rapidly to unimodal visual than to bimodal targets (t(11) = 3.22, p = .008).

2.2.3 Discussion The results of Experiment 2.2 once again demonstrated a significant Colavita effect, even though the participants had to respond to the bimodal stimuli using a dedicated bimodal response key. The participants responded significantly more rapidly and accurately to the unimodal auditory and unimodal visual targets than to the bimodal targets. This result suggests that the relatively high error rates and response latencies seen for the bimodal targets in Experiment 2.1 were not simply caused by the specific response requirements of the task (i.e., having to make two responses rather than just one). Instead, these results suggest that the poorer performance in response to bimodal targets may have resulted from a difficulty

60

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

participants may have had in response selection toward, or the perceptual processing of, the bimodal stimuli. The participants in Experiment 2.2 (in which the three-key response requirement was used) responded more rapidly to the unimodal auditory targets than to the unimodal visual targets (in contrast to Experiment 2.1, in which the two-key response requirement was used). This pattern of response latencies is consistent with that found by others who previously used the three-key response condition (Sinnett et al., 2007). The fact that the Colavita effect still emerged despite this pattern of response latencies, strengthens the argument (outlined earlier; see Section 2.1.3) that the Colavita effect cannot simply be attributable to participants responding to the visual stimuli more rapidly than to the auditory stimuli. It is possible that the reversal in unimodal response latencies (between Experiments 2.1 and 2.2; see Figure 2.3) may have been due to participants delaying their responses to the unimodal visual stimuli in the three-key but not in the two-key response condition. Note that in the two-key response condition, the participants had the option of responding to the visual and auditory components of the bimodal stimulus separately (i.e., if they first responded to the visual component, they could later press the auditory response key). Hence, the participants did not need to delay their responses to visual targets because a visual keypress was the correct response no matter whether a unimodal or bimodal visual target was presented. In the three-key response condition, however, the participants may have delayed their visual keypress responses to unimodal visual stimuli in order to make sure that the target had not been a bimodal stimulus, in which case a visual keypress would have been incorrect.

61

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

800 700 600 500

* *** ***

RT (ms)

400 300 200 100 0 Experiment 2.1 Experiment 2.2

Unimodal auditory targets Unimodal visual targets

* ***

p < .05 p < .001

Figure 2.3. Figure showing the mean RTs (ms) to the unimodal auditory and unimodal visual targets in Experiments 2.1 and 2.2. The error bars indicate the standard errors of the means.

If this explanation were correct, one might have expected the unimodal visual RTs to be slower in Experiment 2.2 than in Experiment 2.1 with the unimodal auditory RTs being no different across the two experiments. This was not the case in our results: Instead, unimodal visual RTs were no slower in Experiment 2.2 than in Experiment 2.1 (mean difference = 4ms; t(23.29) = .10, p = .918), and unimodal auditory RTs were significantly faster in Experiment 2.2 than in Experiment 2.1 (mean difference = 97ms; t(23.93) = 2.26, p = .033). This deviation from the predictions may have been due to the fact that different groups of participants performed each experiment and in addition the stimulus timings were also slightly different. It should, however, be noted that the pattern of response latencies observed goes in the same direction as the predictions.

62

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

2.2.4 Between-participants analysis of Experiments 2.1 and 2.2 If the additional response demands associated with having to make two responses to a bimodal stimulus did make a significant contribution to the Colavita effect observed in Experiment 2.1, then one would expect the magnitude of the effect to have been reduced using the simpler (i.e., unitary) response requirements introduced in Experiment 2.2. This hypothesis was tested directly using a betweenparticipants ANOVA on the bimodal error data with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Response Condition (Two- or Three-response keys). The analysis revealed a significant main effect of Response [F(1, 24) = 17.05, p < .001], with the participants making significantly more visual-only (10.9% of all bimodal trials) than auditory-only responses (3.8% of all bimodal trials), overall. The main effect of Response Condition was, however, not significant (i.e., there was no overall difference in the percentage of bimodal errors made between participants in Experiments 2.1 and 2.2), nor was there any significant interaction between Response and Response Condition, for both terms [F < 1, n.s.]. Thus, the magnitude of the Colavita effect was seemingly unaffected by the nature of the response (Two- or Three-response keys) that participants had to make.

2.2.5 Response biases Finally, it could be argued that the Colavita effect observed in Experiments 2.1 and 2.2 may have been caused by participants being biased to press the visual response key. If this were to have been the case, one would have expected to see participants pressing the visual response key when the visual stimulus was not present (i.e., participants pressing the visual response key on unimodal auditory trials; this 63

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

would be counted as a unimodal auditory error) than pressing the auditory response key when the auditory stimulus was not present11 (a unimodal visual error). This, however, was not the case (see Tables 2.1 and 2.2), indeed the error data revealed that participants in Experiment 2.2 responded somewhat more accurately to unimodal auditory stimuli than to either unimodal visual or bimodal stimuli (i.e., participants pressed the auditory response key when the auditory stimulus was not present more often than they pressed the visual response key when the visual stimulus was not present); this is a pattern that occurs in four of the studies reported in this thesis (Experiments 3.1, 3.2, 7.1, and 7.2) and will be discussed in more detail in Chapter 8. In addition, there was no significant difference between unimodal auditory and unimodal visual error rates in Experiment 2.1 (this pattern of results is also evident in Experiments 3.3, 4.1, 4.2, 4.3, 5.1, and 6.1). These results suggest that the Colavita effect reported here cannot simply be attributed to participants preferentially responding with the visual response key. Further evidence in support of this argument that the Colavita effect does not simply reflect a response bias comes from a study using complex meaningful stimuli (see the study described in Chapter 7, Section 7.4.1). In this study, participants were presented with auditory, visual, or bimodal meaningful stimuli (sounds, images, or both, of cats and dogs) and had to identify the semantic category of the target (i.e., by pressing the cat response key, the dog response key, or both response keys). In other words, on those bimodal target trials in which both a cat and dog were presented (e.g., the sound of a cat, and an image of a dog), the participants had to respond to the semantic categories of the stimuli (e.g., Cat or Dog) rather than to their modality of

11

Remember that it was argued earlier in Section 2.1.2 that the percentage of unimodal auditory errors is indicative of the percentage of trials in which participants responded using the visual response key, and the percentage of unimodal visual errors indicates the percentage of trials in which participants responded by pressing the auditory response key.

64

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

presentation (Auditory or Visual). Nevertheless, the Colavita effect still emerged using these response requirements; that is, on trials where a bimodal stimulus was presented which contained both cat and dog stimuli (one in each modality), participants made more visual-only errors than auditory only errors. This occurred despite the fact that the participants had to respond orthogonally to the sensory modalities under investigation (i.e., there was no longer a dedicated visual response key). These results therefore suggest that the Colavita visual dominance effect emerges regardless of the specific demands of the task, and hence provide further support for the argument that the Colavita effect cannot be explained simply in terms of a response bias to press the visual response key.

2.3

GENERAL DISCUSSION
The primary goal of the experiments reported in this first experimental chapter

was to investigate the contribution of several different factors (response requirements and response biases) to the visual dominance effect originally reported by Colavita (1974). The main finding was that a robust Colavita effect was documented in both experiments; that is, when participants made an error on the bimodal target trials, they failed to respond to the auditory component of the bimodal targets significantly more frequently than they failed to respond to the visual component. It is perhaps worth re-iterating at this point that the magnitudes of the Colavita effects observed in Experiments 2.1 and 2.2 were much smaller than those reported in Colavita’s original studies (see Table 1.1, Chapter 1). Nevertheless, it was in line with the effect sizes seen in more recent studies (Sinnett et al., 2007). This is most likely due to the fact that the response requirements for responding to bimodal targets in the 65

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

present study allowed the participants to respond to both the auditory and visual components of the bimodal targets. It probably also reflects the lack of any deception in these studies (remember that Colavita, 1974, originally made his participants believe that bimodal trials were ‘accidental’). Thus, it is likely that the results reported here provide a more accurate estimate of the actual likelihood that participants will have a difficulty in processing, or responding to, the auditory component of the bimodal stimulus (as compared to the results of the majority of previous studies). Furthermore, it should also be noted that the percentage of bimodal trials that evoked visual-only responses in many previous studies may also have reflected the influence of response biases, or the possibility that participants may have had to suppress their responses to the visual component of the bimodal targets. The second finding to emerge from the present study was that the Colavita effect is not simply attributable to the particular response requirements of the task; that is, a significant Colavita effect was obtained no matter whether participants had to press both the auditory and visual response keys in response to a bimodal target (Experiment 2.1), or whether instead they had a separate bimodal response key (Experiment 2.2) which equated the overall task demands associated with responding to unimodal versus bimodal target events. The results also suggest that the Colavita effect was not caused by participants responding more rapidly to visual targets than to auditory targets, as the Colavita effect still emerged in Experiment 2.2, in which the responses to auditory targets were significantly more rapid than responses to visual targets. In sum, the results of the two experiments reported here demonstrate that the Colavita effect is perceptually-based effect, one that cannot simply be explained in terms of response biases, or the relative response latencies to unimodal auditory and visual stimuli.

66

CHAPTER 2 - RESPONSE REQUIREMENTS AND RESPONSE BIASES

The bimodal targets in Experiments 2.1 and 2.2 were presented less frequently than the unimodal targets. The ratio of stimulus presentation was 40A:40V:20AV (this is true of many of the previous studies on the Colavita effect; Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979; Egeth & Sager, 1977; Johnson & Shapiro, 1989; Quinlan, 2000; Shapiro et al., 1984; Sinnett et al., 2007). While the results of Experiments 2.1 and 2.2 have demonstrated that the Colavita visual dominance effect reported here can not be attributed to response biases, or the relative response latencies, it could be argued that the difficulty that participants had in responding to bimodal targets may have been caused solely by the relative infrequency of those targets. This issue will therefore be addressed in Chapter 3, in a series of experiments in which the relative probabilities of auditory, visual, and bimodal targets will be varied.

67

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

CHAPTER 3
3.0 THE ROLE OF BIMODAL STIMULUS PROBABILITY
The majority of the previous studies of the Colavita effect have presented the auditory, visual, and bimodal stimuli in the ratio of 40A:40V:20AV (similar to the proportions that Colavita, 1974, used in his original study). However, as the likelihood of stimulus repetitions occurring on consecutive trials is higher for stimuli that occur with a higher probability within a given block of trials, this means that unimodal auditory and unimodal visual targets would be more likely than bimodal targets to benefit from any stimulus repetition effects1. The literature on repetition effects (e.g., Bertelson, 1961; Kornblum, 1973; see also Soetens, Melis, & Notebaert, 2002, for a more recent review) shows that when the same stimulus is presented to participants on successive trials in a speeded discrimination task setting, responses tend to be faster and more accurate than on trials in which no such repetition occurs (this is known as repetition priming; Campbell & Proctor, 1993; Pashler & Baylis,

Remember that it was argued in Section 2.1.4 (Chapter 2) that participants tend to respond to bimodal stimuli using a single act of response selection to the target attribute of bimodality (a response couplet). Therefore, responses to bimodal stimuli (i.e., pressing both the auditory and visual response keys) are distinct from responses to unimodal stimuli (i.e., pressing only the auditory response key, or only the visual response key). Any stimulus-response repetition benefits should, hence, only occur for the repetition of identical stimuli (e.g., a bimodal target preceded by another bimodal target) rather than for stimulus repetitions that only share certain attributes but not others (e.g., a bimodal target preceded by a unimodal auditory or unimodal visual target, or a unimodal visual or auditory target preceded by a bimodal target).

1

68

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

1991; Soetens, 1998), and it is thought that this facilitation in responding to repeated stimuli is due to the speed-up of stimulus perceptual processing and response selection. Furthermore, Spence et al. (2001a) have shown that RT benefits for the most frequently presented stimulus (or modality) in a block of trials are at least partly due to such repetition priming effects. Responses to bimodal targets would be less likely to profit from any stimulusresponse repetition effects that may have benefited performance for the more frequently presented unimodal targets. Hence, it is possible that the lower accuracy and slower response latencies that have been observed in response to bimodal targets than to either unimodal auditory or unimodal visual targets could be explained by the low frequency at which the bimodal targets have typically been presented in most studies of the Colavita effect (as well as in Experiments 2.1 and 2.2), rather than necessarily being caused by a difficulty that participants may have had in the processing of, or response selection to, bimodal targets. The fact that participants in Experiment 2.1 responded less accurately to bimodal targets preceded by unimodal auditory or unimodal visual targets, than to bimodal targets preceded by bimodal targets, supports this notion that stimulus repetition effects do contribute to participants’ performance on the Colavita task (see Chapter 2, Section 2.1.4). Experiment 3.1 was therefore designed to investigate whether the impaired performance that has typically been observed on bimodal trials in studies of the Colavita effect is due to the difficulty that participants have had in responding to both components of the bimodal target stimuli, or whether instead it is due to the lower frequency with which the bimodal targets have often been presented. The design was identical to that used in Experiment 2.1 with the sole exception that equal numbers of auditory, visual, and bimodal target trials were now presented in each block of

69

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

experimental trials. If the low probability of bimodal trials contributes significantly to the Colavita effect, one would expect to observe a significantly smaller Colavita effect in Experiment 3.1 (33A:33V:33AV) than in Experiment 2.1 (40A:40V:20AV).

3.1

EXPERIMENT 3.1

3.1.1 Methods Participants. 14 naïve participants (mean age of 20 years, age range from 1829 years; 6 males and 8 females) took part in Experiment 3.1. All of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes. Apparatus, materials, design, and procedure. These were exactly the same as in Experiment 2.1 with the sole exception that each block of trials now consisted of an equal number (33) of unimodal visual, unimodal auditory, and bimodal targets.

3.1.2 Results Error data. The participants failed to respond on 0.9% of the trials overall, and these trials were not included in the data analyses. The data from Experiment 3.1 are shown in Table 3.1. An ANOVA performed on the data from the bimodal trials in which the participants failed to respond correctly with the factor of Response (Auditory-only or Visual-only), revealed a significant main effect [F(1, 13) = 25.98, p < .001]. The participants made significantly more visual-only than auditory-only responses (6.6% vs. 2.0% of all bimodal trials, respectively). Thus, a significant Colavita effect emerged despite the fact that the stimulus probabilities were now equalised. 70

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

An ANOVA performed on the error data from the unimodal and bimodal trials revealed a significant main effect of Target Stimulus (Auditory, Bimodal, or Visual) [F(2, 26) = 3.56, p = .043], attributable to the participants responding more accurately on the unimodal auditory trials (5.1% errors) than on either the bimodal (8.6% errors; t(13) = 2.47, p = .028) or the unimodal visual trials (7.5% errors; t(13) = 3.81, p = .002), but not significantly more accurately on the unimodal visual trials than on the bimodal trials (t(13) = .48, p = .640).
Table 3.1. Mean error rates for the unimodal auditory, unimodal visual, and bimodal target stimuli in Experiment 3.1. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. Standard errors are shown in parentheses.

Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses

5.1 (0.8) 7.5 (1.0)

2.0 (0.5) 6.6 (1.2)

RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses

580 (21) 530 (20)

609 (21) 576 (23)

RT data. The RT data from those trials in which the participants made a correct response were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), and Target Type (Unimodal or Bimodal). The analysis revealed a significant main effect of Target Type [F(1, 13) = 20.90, p = .001], attributable to the participants responding significantly more rapidly to unimodal (555ms) than to bimodal targets (593ms). Note that this is the same pattern of results as in Experiment 2.1. There was also a significant main effect of Target Modality [F(1, 13) = 22.44, p < 71

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

.001], with the participants responding significantly more rapidly to visual (553ms) than to auditory targets (594ms), once again, replicating the results of Experiment 2.1. Note, however, that there appears to have been a small speed-accuracy trade-off in participants’ responses to the unimodal auditory and unimodal visual targets. The interaction between Target Modality and Target Type was, however, not significant [F(1, 13) = 3.00, p = .107].

3.1.3 Discussion The results of Experiment 3.1 revealed a significant Colavita visual dominance effect; that is, when the participants failed to respond correctly on the bimodal target trials (which they did on 8.6% of all bimodal trials), they made significantly more visual-only than auditory-only responses (6.6% vs. 2.0% of all bimodal trials, respectively). Furthermore, the participants in Experiment 3.1 still responded significantly more rapidly, and somewhat more accurately, to unimodal than to bimodal targets, even though there were now equal numbers of all three types of stimulus (cf. Experiment 2.1). The results of Experiment 3.1 therefore refute the argument that the pattern of poorer performance on the bimodal trials (in terms of the RTs and error rates) than on unimodal trials, observed in Experiments 2.1 and 2.2 (and in the previous studies of the Colavita effect which presented participants with a low probability of bimodal trials) simply reflected a strategy adopted by participants to preferentially monitor those stimuli that had a higher probability of occurrence (cf. Spence et al., 2001a). Rather, it can instead be argued that the relatively poor performance on the bimodal trials may have resulted from participants’ difficulty in perceptually processing, or selecting a response toward, the bimodal targets.

72

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

3.1.4 Between-experiments analysis of Experiments 2.1 and 3.1 The next step was to explore what effect, if any, the relative frequency of unimodal and bimodal targets (the ratio of 40A:40V:20AV, as in Experiment 2.1 vs. the ratio of 33A:33V:33AV, as in Experiment 3.1) had on the magnitude of the Colavita effect. A between-participants ANOVA was therefore conducted on the bimodal error data, with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Experiment (Experiment 2.1 vs. Experiment 3.1). This analysis revealed a significant main effect of Response [F(1, 26) = 15.19, p = .001], with the participants making significantly more visual-only (9.4% of all bimodal trials) than auditory-only responses (3.2% of all bimodal trials) overall, as expected. Participants made more bimodal errors in Experiment 2.1 (16.5% of all bimodal trials) than in Experiment 3.1 (8.6% of all bimodal trials), resulting in a significant main effect of Experiment [F(1, 26) = 4.91, p = .036]. Importantly, however, there was no interaction between Response and Experiment [F < 1, n.s.], thus showing that the magnitude of the Colavita effect was not significantly affected by the change in the relative probability of occurrence of the bimodal stimuli from 20% to 33% (though note that the size of the Colavita effect did decrease numerically as bimodal stimulus frequency was increased). In summary, the results of the between-participants analysis of Experiments 2.1 and 3.1 demonstrated that the magnitude of the Colavita effect was unaffected by equalizing the stimulus probabilities. Hence, the relatively low frequency with which the bimodal targets were presented in Experiments 2.1, 2.2, and in the previous studies of the Colavita effect (i.e., 40A:40A:20AV) does not appear to contribute significantly to the magnitude of the Colavita effect, at least when compared with a condition in which the three types of targets occur with an equal probability.

73

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

However, it is important to note that participants made fewer errors in responding to bimodal targets when the stimulus probabilities were equalised (as compared to when the bimodal targets were presented relatively infrequently). This suggests that the relative frequency with which the unimodal auditory, unimodal visual, and bimodal target stimuli are presented does play at least some role in causing the poorer performance reported on bimodal trials. Therefore, Experiment 3.2 was conducted in order to investigate whether increasing the frequency of bimodal targets so that they constituted the most common type of target would eliminate (or at least reduce) the magnitude of the Colavita effect (due to an improvement in the accuracy of responding to the bimodal trials). The design was identical to that used in Experiment 2.1 with the sole exception the auditory, visual, and bimodal targets were presented in the ratio of 20A:20V:60AV (compared to 40A:40V:20AV as in Experiment 2.1). If the Colavita effect reflects a robust empirical phenomenon then participants should make more visual-only than auditory-only responses on the bimodal trials, despite the high percentage of bimodal targets.

3.2

EXPERIMENT 3.2

3.2.1 Methods Participants. 18 naïve participants (mean age of 22 years, age range from 1829 years; 6 males and 12 females) took part in Experiment 3.2. All except one of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes. Apparatus, materials, design, and procedure. These were exactly the same as in Experiment 2.1 with the exceptions that the ISI was now 1500ms (compared to

74

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

1800ms, as in Experiment 2.1), and there were 20 visual, 20 auditory, and 60 bimodal trials in each block.

3.2.2 Results Error data. The participants failed to respond on 1.2% of the trials overall, and these trials were not included in the data analyses. The data from Experiment 3.2 are shown in Table 3.2. An ANOVA performed on the data from the bimodal trials in which the participants failed to respond correctly with the factor of Response (Auditory-only or Visual-only), revealed a significant main effect [F(1, 17) = 8.04, p = .011]. Participants made significantly more visual-only than auditory-only responses (1.5% vs. 0.3% of all bimodal trials, respectively), thus demonstrating a very small (but nevertheless still significant) Colavita effect.
Table 3.2. Mean error rates for the unimodal auditory, unimodal visual, and bimodal target stimuli in Experiment 3.2. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. Standard errors are shown in parentheses.

Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses

5.8 (1.3) 9.9 (1.0)

0.3 (0.1) 1.5 (0.4)

RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses

620 (24) 596 (22)

527 (21) 509 (21)

75

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

An ANOVA performed on the error data from the unimodal and bimodal trials revealed a significant main effect of Target Stimulus (Auditory, Bimodal, or Visual) [F(2, 34) = 28.42, p < .001], attributable to participants responding more accurately on the bimodal target trials (1.8% errors) than on either unimodal auditory (5.8% errors; t(17) = 3.95, p = .001) or unimodal visual trials (9.9% errors; t(17) = 8.08, p < .001), and more accurately on the unimodal auditory than on the unimodal visual target trials (t(17) = 3.42, p = .003). RT data. The RT data from those trials in which the participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual) and Target Type (Unimodal or Bimodal). The analysis revealed a significant main effect of Target Type [F(1, 17) = 66.28, p < .001], attributable to participants responding significantly more rapidly on the bimodal (518ms) than on the unimodal target trials (608ms). There was also a significant main effect of Target Modality [F(1, 17) = 25.55, p < .001], with participants responding more rapidly to visual (552ms) than to auditory targets (573ms). The interaction between Target Modality and Target Type was not significant [F < 1, n.s.].

3.2.3 Discussion When the majority of trials were bimodal, participants responded both more rapidly and more accurately on the bimodal trials than on the unimodal trials. This pattern of results is the reverse of what has been reported in the majority of the previous studies of the Colavita effect (when the majority of trials were unimodal). Nevertheless, despite this pattern of results being obtained, a very small Colavita effect was still observed in Experiment 3.2 (that is, the Colavita effect was not eliminated completely). When the participants failed to respond correctly on the 76

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

bimodal target trials (which they did on 1.8% of all bimodal trials), they made significantly more visual-only than auditory-only responses (1.5% vs. 0.3% of all bimodal trials, respectively).

3.2.4 Between-experiments analysis of Experiments 3.1 and 3.2 Next, the effect of increasing the relative frequency of the bimodal targets (from 33A:33V:33AV in Experiment 3.1 to 20A:20V:60AV in Experiment 3.2) on the magnitude of the Colavita effect was investigated. The bimodal error data were analysed using a between-participants ANOVA with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Experiment (Experiment 3.1 vs. Experiment 3.2). The analysis revealed a significant main effect of Response [F(1, 30) = 39.39, p = .003], with participants making more visual-only than auditory-only responses (4.0% vs. 1.2% of all bimodal trials), overall as expected. There was also a significant main effect of Experiment [F(1, 30) = 18.89, p < .001], attributable to participants making more bimodal errors in Experiment 3.1 than in Experiment 3.2 (4.3% vs. 0.9% of all bimodal trials). Crucially, this analysis also revealed a significant interaction between Response and Experiment [F(1, 30) = 14.27, p = .001], attributable to the Colavita effect being significantly larger in Experiment 3.1 than in Experiment 3.2 (4.6% vs. 1.1%; t(18.18) = 3.50, p = .003). Hence, the results of this between-experiments comparison show that the Colavita effect was significantly reduced, but not eliminated, by increasing the bimodal target frequency so that they constituted the most frequent target probability. As increasing the probability of bimodal targets appears to reduce the magnitude of the Colavita effect, Experiment 3.3 was conducted to explore whether it is possible to eliminate the Colavita effect altogether using an even more extreme manipulation of 77

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

stimulus probability. In Experiment 3.3, the same group of participants was presented with blocks of trials in which bimodal targets were presented at the frequencies of 90%, 50%, and 10% (high, medium, and low bimodal probability conditions, respectively). Based on the results of Experiments 3.1 and 3.2, one might predict that the Colavita effect would be eliminated in the high, but not the medium or low bimodal probability conditions.

3.3

EXPERIMENT 3.3

3.3.1 Methods Participants. 12 naïve participants (mean age of 22 years, age range from 1927 years; 4 males and 8 females) took part in Experiment 3.3. All except one of the participants were right-handed by self-report. The experimental session lasted for approximately 60 minutes. Apparatus, materials, design, and procedure. These were exactly the same as in Experiment 2.1 with the exceptions that the ISI was now 1500ms (compared to 1800ms, as in Experiment 2.1), and the participants were now presented with 9 blocks of 100 trials. These consisted of a set of three high bimodal probability blocks (5A:5V:90AV), a set of three medium bimodal probability blocks (25A:25V:50AV), and a set of three low bimodal probability blocks (45A:45V:10AV). The order in which the high, medium, and low bimodal probability sets of blocks were presented was counterbalanced across participants.

78

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

3.3.2 Results Error data. The participants failed to respond on 0.6% of the trials overall, and these trials were not included in the data analyses. The data from Experiment 3.3 are shown in Table 3.3.
Table 3.3. Mean error rates for the unimodal auditory, unimodal visual, and bimodal target stimuli in the high, medium, and low bimodal probability conditions in Experiment 3.3. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. Standard errors are shown in parentheses. Bimodal probability Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses High Medium Low

53.0 (5.4) 37.9 (6.4)

16.6 (2.7) 12.9 (2.3)

7.3 (2) 7.6 (1.4)

0.7 (0.2) 0.9 (0.3)

2.3 (1.1) 4.3 (1.6)

6.8 (2.3) 11.5 (3.0)

RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses

581 (43) 565 (36)

496 (26) 512 (31)

420 (30) 427 (32)

313 (19) 302 (20)

494 (24) 462 (24)

522 (31) 512 (30)

The data from the bimodal trials in which the participants failed to respond to one of the two stimuli were analysed using an ANOVA with the factors of Response (Auditory-only or Visual-only) and Bimodal Probability (High, Medium, or Low). This analysis revealed a significant main effect of Bimodal Probability [F(1.36, 14.99) = 10.18, p = .009], attributable to participants making significantly more bimodal errors in the low bimodal probability condition (9.1% errors) than in the medium (3.3% errors; t(11) = 2.81, p = .017) or high (0.8% errors; t(11) = 3.28, p = .007) bimodal probability conditions, with the difference between the latter two conditions just failing to reach statistical significance (t(11) = 1.98, p = .073). There 79

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

was also a significant main effect of Response [F(1, 11) = 10.18, p = .009], with participants making more visual-only than auditory-only responses (5.6% vs. 3.3% of all bimodal trials, respectively) overall, thus demonstrating a significant Colavita effect.
16 14 12 10 8 6 4 2 0 High Medium Bimodal probability Low

*

Errors (%)

*

Auditory-only Visual-only

*

p < .05

Figure 3.1. Figure showing the percentages of auditory-only and visual-only errors in the high, medium, and low bimodal probability conditions in Experiment 3.3. The error bars indicate the standard errors of the means.

Finally, there was a significant interaction between Response and Bimodal Probability [F(1.16, 12.79) = 4.11, p = .031]. This interaction can be explained in terms of a significant Colavita effect being present in the low and medium bimodal probability conditions (4.7% and 2.0%, respectively; t(11) = 2.50, p = .030; t(11) = 2.94, p = .013), but not in the high bimodal probability condition (0.2%; t(11) = 0.81, p = .435; see Figure 3.1). In fact, the Colavita effect was significantly larger in the low and medium bimodal probability conditions than in the high bimodal probability conditions (t(11) = 2.40, p = .035; t(11) = 2.96, p = .013). However, the Colavita

80

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

effect was numerically, but not significantly, larger in the low than in the medium bimodal probability condition (t(11) = 1.42, p = .183). An ANOVA performed on the error data with the factors of Target Stimulus (Auditory, Bimodal, or Visual) and Bimodal Probability (High, Medium, or Low) revealed a significant main effect of Target Stimulus [F(2, 22) = 24.98, p < .001], attributable to participants making significantly fewer errors in response to the bimodal targets (8.8% errors) than to the unimodal auditory2 (25.6% errors; t(11) = 5.24, p < .001) or to the unimodal visual targets (19.4% errors; t(11) = 2.77, p = .018), overall, just as in Experiment 3.2. However, the participants made no more errors on unimodal auditory than on unimodal visual trials (t(11) = 1.99, p = .072). The analysis also revealed a significant main effect of Bimodal Probability [F(1.87, 20.55) = 12.50, p < .001], attributable to participants making more errors in the high bimodal probability (30.8% errors) than in the medium or low bimodal probability conditions (12.0% and 11.0% errors, respectively; t(11) = 5.51, p < .001; t(11) = 5.00, p < .001), but no more errors in the medium than in the low bimodal probability condition (t(11) = .64, p = .538). Finally, there was a significant interaction between Target Stimulus and Bimodal Probability [F(4, 44) = 36.39, p < .001]. Participants responded more accurately to bimodal than to unimodal auditory or unimodal visual targets in the high (mean difference = 51.3% and 36.2%, respectively; t(11) = 9.84, p < .001; t(11) = 5.56, p < .001), and medium bimodal probability conditions (mean difference = 10.0% and 6.2%, respectively; t(11) = 3.30, p = .007; t(11) = 2.24, p = .046). By contrast, in the low bimodal probability conditions, participants responded less accurately to bimodal than to unimodal auditory or unimodal visual targets (mean

The unimodal target error rates were high in the high bimodal probability condition due to the fact that unimodal auditory and visual targets each constituted only 5% of the total number of trials presented. Hence, any errors on those trials would result in a large percentage of errors.

2

81

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

difference = 10.9% and 10.7%, respectively; t(11) = 2.32, p = .041; t(11) = 2.13, p = .057). RT data. The RT data from those trials in which the participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), Target Type (Unimodal or Bimodal), and Bimodal Probability (High, Medium, or Low). The analysis revealed a significant main effect of Target Type [F(1, 11) = 26.81, p < .001], attributable to participants responding significantly more rapidly on bimodal (434ms) than on unimodal target trials (500ms) overall. Participants responded less rapidly in the medium bimodal probability (491ms) than in either the high or low bimodal probability conditions (440ms and 470ms, respectively; t(11) = 4.32, p = .001; t(11) = 2.39, p = .036), but no more rapidly in the high than in the low bimodal probability condition (t(11) = 1.09, p = .298), resulting in a significant main effect of Bimodal Probability [F(2, 22) = 5.84, p = .009]. Finally, there was also a significant interaction between Target Type and Bimodal Probability [F(2, 22) = 110.21, p < .001], with participants responding more rapidly to the unimodal than to the bimodal targets in the low bimodal probability condition (424ms vs. 517ms; t(11) = 10.37, p < .001), whereas the reverse was true in the high bimodal probability condition (573ms vs. 308ms; t(11) = 9.09, p < .001). This comparison between unimodal and bimodal RTs just failed to reach statistical significance for the medium bimodal probability condition (504ms vs. 478ms; t(11) = 2.04, p =.067). None of the other terms reached significance: for Target Modality [F(1, 11) = 2.20, p = .166], for Target Modality × Target Type [F(1, 11) = 3.74, p = .079], and for all other terms [F < 1, n.s.].

82

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

3.3.3 Discussion The Colavita effect emerged in the low and medium bimodal probability conditions (4.7% and 2.0%), as predicted, but was eliminated in the high bimodal probability condition (0.2%). Therefore, the results of Experiment 3.3 show that the Colavita effect can be eliminated by extreme manipulations of stimulus probability, such as when the bimodal targets are presented on 90% of the trials. The fact that the magnitude of the Colavita effect itself decreased as the bimodal stimulus probability increased (and the accuracy and latencies of participants’ responses improved), suggests that those factors that improve the speed and accuracy of participants’ responses to bimodal targets, may contribute to the elimination of the Colavita effect.

3.4

GENERAL DISCUSSION
In summary, the Colavita effect was present when bimodal targets constituted

10%, 33%, 50%, and 60% of the trials (Colavita effect = 4.7%, 4.6%, 2.0%, and 1.2%, respectively) but was eliminated when 90% of the trials were bimodal (Colavita effect = 0.2%). One possible explanation of these results is in terms of the Supervisory Attentional System (SAS; Norman & Shallice, 1980; Shallice, 1988, also see Stuss, Shallice, Alexander, & Picton, 1995), which has been posited to help understand the sustained attention state. According to proponents of the SAS, the execution of responses (referred to as schema) is determined by their level of activation relative to their competitors (see Manly, Robertson, Galloway, & Hawkins, 1999, for a review). The level of activation of the response schema can be determined either by endogenous control or by externally-driven (i.e., exogenous) input (such as the presentation of a target itself). It can also be determined by the strength of the

83

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

association between the external inputs and the responses. Thus, when a stimulus is presented frequently, the frequent presentation of the stimulus itself serves as exogenous support to maintain the level of activation of the response schema. In contrast, when a stimulus is presented less frequently, the activation level of the response schema must be maintained by endogenous attention for the appropriate response selection. Explaining the results of Chapter 3 within the SAS framework, when bimodal targets are presented infrequently, the schema mediating the bimodal responses3 is deprived of the external input (of the bimodal target; i.e., exogenous support), and hence will have a low level of activation. Thus, when bimodal targets are presented infrequently, attention directed endogenously is needed to maintain the activation level of the bimodal response schema (to overcome it being at a low level of activation), so that the bimodal responses can be executed when a bimodal target is actually presented. In contrast, when bimodal targets are presented more frequently, the bimodal response schema would be exogenously driven by the presentation of the bimodal targets themselves (because it would be activated frequently by the bimodal targets). Hence, there would be a reduced reliance on endogenous control for maintaining the activation level of the bimodal response schema. The worse performance (in terms of response latencies and accuracy) observed on bimodal trials when the bimodal targets were presented infrequently (compared to when they were presented relatively frequently) would therefore reflect the difficulty that participants had in maintaining the bimodal response schema.

Remember that it was argued in Section 2.1.4 (Chapter 2) that participants tend to respond to both modalities of a bimodal stimulus as if they constituted a single audiovisual target using a single act of response selection. Therefore, in the context of the SAS framework, the schema for a bimodal response schema would be a single paired response. Hence, one possible effect of decreasing the bimodal stimulus probability might also be to decrease the occurrence of response coupling, thus leading to a larger Colavita effect.

3

84

CHAPTER 3 - BIMODAL STIMULUS PROBABILITY

The results of the experiments reported in Chapter 3 therefore suggest two points. First, the fact that participants’ performance on bimodal trials was modulated by the relative frequency of the bimodal targets suggests that the poorer performance observed on bimodal trials may be a consequence of the difficulty people may have in sustaining the response schema. Second, the finding that the Colavita effect still emerged when bimodal targets were presented more frequently (i.e., when bimodal targets were presented on 10%, 33%, 50%, and 60% of the trials) suggests that participants also have a difficulty in the response selection toward, or the perceptual processing of, both components of the bimodal stimulus (rather than being caused by the relative infrequency of presentation of the bimodal targets). Whilst the results of Experiments 3.1-3.3 suggest that the Colavita effect may be caused by a difficulty in sustaining the bimodal response schema, it is still uncertain as to whether the Colavita effect is also the result of participants tending to direct their attention endogenously toward the visual modality (as claimed by Colavita & Weisberg, 1979; Egeth & Sager, 1977; Posner et al., 1976). This is due to the many confounding factors present in the early research investigating those claims (as mentioned in Chapter 1, Sections 1.4.2 and 1.5). The first experiment reported in Chapter 4 was therefore designed to investigate whether directing a participant’s attention endogenously toward the visual or auditory modalities can modulate the Colavita visual dominance effect, after which the ability of attention directed exogenously to modulate the Colavita effect will be explored.

85

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

CHAPTER 4
4.0 ATTENTION AND THE COLAVITA EFFECT
In Posner et al.’s (1976) oft-cited article on visual dominance and its origins, it was proposed that visual dominance might represent a by-product of attentional processes. In particular, it was hypothesised that humans have a tendency to endogenously attend toward the visual modality in order to compensate for the poorer alerting properties of the visual system (as compared to the auditory or tactile systems; e.g., Klein, 1977; Spence et al., 2001a; Whipple et al., 1899). According to this account, therefore, the Colavita effect occurs because participants endogenously direct their attention toward the visual modality when bimodal stimuli are presented. Therefore, unless an alerting effect (elicited by the presentation of an auditory stimulus) exogenously drew attention toward the auditory modality, or unless the participants endogenously directed their attention toward the auditory modality, the participants’ attention would remain focused on the visual modality with the result that the visual component of a bimodal stimulus would be processed more readily than the auditory component. It should be noted here that, in line with the majority of existing theories on attention, attention is thought of as a resource which can be either controlled voluntarily and with effort by the participant (i.e., endogenously directed, top-down, 86

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

or goal driven attention), or it can be captured automatically and transiently by an external event (i.e., exogenously directed, bottom-up, or stimulus-driven attention; Arvidson, 2003; Chun & Wolfe, 2001; Posner, 1980; Yantis, 1998). Following on from Posner et al.’s (1976) proposal, a number of researchers, including Posner et al. themselves, attempted to explain the Colavita effect in terms of this attentional account of visual dominance (Colavita & Weisberg, 1979; Egeth & Sager, 1977), arguing that this bias to attend endogenously toward the visual modality may have been the cause of participants’ failure to respond to the auditory stimulus on the bimodal trials. Support for the argument that the modality toward which attention is endogenously directed can modulate the Colavita effect came from subsequent research reported by Egeth and Sager (1977), Quinlan (2000), and Sinnett et al. (2007). In Egeth and Sager’s (1977) study, participants were instructed to press the tone-key in response to both unimodal auditory and bimodal stimuli, and the light-key in response to unimodal visual stimuli. The slowing of RTs to bimodal stimuli (using the tone key) relative to RTs to the unimodal auditory stimuli (also using the tone key) was interpreted as indicating that the visual stimulus had ‘interfered’ with (i.e., dominated over) the processing of the auditory stimulus (see Chapter 1, Section 1.4.2). Egeth and Sager performed two types of attentional manipulations (a probability manipulation, and an instructional manipulation) which both appeared to modulate the ‘interference’ (slowing of bimodal RTs relative to unimodal auditory RTs) by the visual stimulus during the bimodal trials. In Egeth and Sager’s (1977, Experiment 3) probability manipulation experiment, they manipulated the relative probabilities of the bimodal and visual targets (40A:40V:20AV to 40A:20V:40AV, for the Low and High bimodal

87

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

probability conditions) and found that the ‘interference’ effect was greater in the Low (interference = 57ms) than in the High bimodal probability condition (in which bimodal RTs were actually 18ms faster than unimodal auditory RTs; see Figure 4.1).

*
500

***
450

RT (ms)

Unimodal auditory targets Bimodal targets 400

350 High bimodal probability Low bimodal probability Expect audition Control

Experiment 3 Probability manipulation

Experiment 5 Instructional manipulation

* ***

p < .05 p < .001

Figure 4.1. Graph showing the ‘interference’ effects in the High and Low bimodal probability conditions (where the ratios of the stimuli were 40A:40V:20AV and 40A:20V:40AV, respectively; Experiment 3), and in the Expect audition and Control conditions (where participants were either told to “attend to their ears”, or not given any such instructions; Experiment 5), in Egeth and Sager’s (1977) study.

In Egeth and Sager’s (1977; Experiment 5) instructional manipulation, they manipulated the task instructions; participants in the Expect audition condition were instructed to “attend to their ears” (participants were told that they should do this because the majority of their responses would be tone responses), and participants in the Control condition were given no instructions about which sensory modality they should attend to. As expected, the ‘interference’ effect was smaller in the Expect audition condition (interference = 17ms) than in the Control condition (interference = 51ms). Using similar criteria for the measurement of visual dominance, other 88

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

researchers have found that visual ‘interference’ on auditory processing is only present when there is a higher proportion of visual than bimodal trials (see Quinlan, 2000). These findings have led many researchers to conclude that the Colavita effect is caused by participants preferentially directing their attention endogenously toward the visual modality under normal conditions (Colavita & Weisberg, 1979; Egeth & Sager, 1977). The most appropriate interpretation of the results reported by both Egeth and Sager (1977) and Quinlan (2000), however, remains open to question. First, the somewhat unusual requirement that participants should press the tone key in response to the bimodal targets may have caused response conflicts to arise, due to participants presumably having to suppress any tendency that they may have had to respond to the visual component of the bimodal targets (as discussed in Chapter 1, Section 1.4.2). In addition, the fact that the auditory stimuli were presented from a different position (over headphones) than the visual stimuli (which were presented from in front of the participant) may have meant that the probability and instructional manipulations simply resulted in a manipulation of the participants’ spatial attention, rather than a shift of their attention toward a particular sensory modality (see Spence & Driver, 1997b, on this point). Finally, the facilitation of the bimodal RTs on the High bimodal probability condition, which arose as a result of increasing the probability with which the bimodal stimuli were presented, may also reflect some element of repetition priming (see Spence et al., 2001a; see also Chapter 3). Thus, the decreased ‘interference’ observed in the RT data in the High bimodal probability conditions (as well as in the Expect audition condition) in Egeth and Sager’s (1977) study may have been at least partly attributable to some unknown combination of response conflicts, manipulations of spatial attention, and/or repetition

89

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

priming effects; that is, we cannot be certain that it was necessarily caused by a decrease in visual dominance per se. The evidence provided by Egeth and Sager (1977) and Quinlan (2000) is therefore, at best, equivocal in terms of determining whether manipulations of endogenous attention can modulate the Colavita effect. In order to investigate whether directing attention endogenously toward a particular modality can modulate the Colavita effect, Sinnett et al. (2007) performed a study in which they manipulated the relative probabilities with which the stimuli were presented. To avoid the potential confounds present in the earlier studies, Sinnett et al. used a different type of response requirement than that used by Egeth and Sager (1977) and, in addition, presented the stimuli from the same spatial location. Sinnett et al. (Experiment 5) presented their participants with a rapid stream of complex audiovisual stimuli, and participants had to respond to pre-specified auditory, visual, and bimodal targets (a picture of a traffic light, the sound of a cat meowing, or both) which were presented occasionally in the stream (there were 50 different stimuli in each stream, consisting of 1 target and 49 distractors). In contrast to Egeth and Sager (1977), Sinnett et al. defined the Colavita effect as occurring when participants made significantly more visual-only than auditory-only errors (i.e., this is the same definition of the Colavita effect which is used throughout this thesis). They found that manipulating the relative probabilities with which the auditory, visual, and bimodal stimuli were presented (60A:20V:20AV vs. 20A:60V:20AV, in the Expect audition and Expect vision conditions, respectively) significantly modulated the Colavita effect. In particular, the participants made more auditory-only than visual-only responses in the Expect audition condition (see Figure 4.2; although note that this trend failed to reach statistical significance).

90

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Sinnett et al. (2007), however, noted that while it is generally assumed that probability manipulations bias the direction of attention to the most frequent stimulus type (because participants come to expect stimuli from the most frequent stimulus type to be presented), probability manipulations may also potentially introduce a bias in response mechanisms. Specifically, they may prime participants to make the particular responses associated with the most frequent stimulus type, thereby confusing attentional manipulation effects with a response bias effect. Therefore, in order to get around the problems associated with stimulus probability manipulations, Sinnett et al. (2007, Experiment 6) investigated whether an attentional manipulation, which did not involve altering the relative probabilities with which participants had to make the various responses, would also modulate the Colavita effect. Sinnett et al. tested this by varying the auditory or visual perceptual load of the task. According to perceptual load theory (e.g., Lavie, 2005), the amount of attentional resources available to process information at any given time is dependent on the perceptual load required by the task being performed. Sinnett et al. (2007) decreased the perceptual load of the auditory or visual streams by decreasing the variability amongst the irrelevant distractors presented in that stream (i.e., presenting only 3 rather than 49 different distractors in the audiovisual stream). They hypothesised that decreasing the auditory load (while keeping the visual load high) should increase the attentional resources left over to process the auditory targets, and therefore decrease the visual dominance observed (and vice versa for the visual modality). It should be noted that in making this hypothesis, Sinnett et al. (2007) assumed that attentional capacity is not shared

91

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

between the sensory modalities (see Rees, Frith, & Lavie, 2001, which suggests that attentional capacity is not shared between modalities)1. While Sinnett et al. (2007) found that the Colavita effect was significantly smaller in the Low auditory load condition than in the Low visual load condition, the Colavita effect was nevertheless still significant in both conditions (see Figure 4.2). Thus, Sinnett et al. concluded that manipulations of attention, by varying the perceptual load of the task (to ‘free up’ auditory/visual attentional resources) or by changing the relative probabilities of the stimuli, can modulate, but not significantly reverse the Colavita effect. It should be noted here that in comparing the results of Experiments 5 and 6, Sinnett et al. (2007) noticed that both the modulation of the Colavita effect, and the RT advantage for the modality toward which participants were supposed to be attending, were greater in Experiment 5 (probability manipulation) than in Experiment 6 (perceptual load manipulation). They therefore argued that response biases to respond to targets presented in the most frequent modality had exaggerated the effect that their stimulus probability manipulations had seemingly had on the Colavita effect.

1

While this appears to contradict the premise that the Colavita effect emerges when attention is directed toward the visual modality (i.e., the assumption is that attentional capacity is shared between modalities), it should be noted that there is both evidence suggesting that attentional resources are shared between modalities and evidence suggesting the opposite. First, there is a wealth of evidence in the literature suggesting that attention is shared between modalities (Spence & Driver, 1995, 1996, 1997a, b). Indeed, other studies investigating perceptual load (e.g., Berman & Colby, 2002; Houghton, Macken, & Jones, 2004) have found evidence supporting the notion that attentional capacity is shared between modalities. Second, studies that have investigated whether attentional capacity is shared between modalities using perceptual load have tended to use continuous tasks - such as monitoring an auditory word stream for bisyllabic words while being presented with a visual motion display (e.g., Rees et al., 2001). Such tasks (i.e., passively viewing a visual motion display) can occur automatically and without discrete decision making (i.e., making a response). Thus, while it could be argued that while Rees et al.’s results appear to show that continuous visual sensory processes can occur when attention is directed toward the auditory modality, Rees et al.’s results do not show that participants’ making of discrete decisions to respond to visual stimuli would not be impaired if they were attending to the auditory modality. Third, tasks that do require the discrete decision making involved in the Colavita effect (such as in the Attentional Blink paradigm, as will be discussed in Chapter 9, Section 9.2.1.2) appear to support the notion that attentional resources do appear to be shared between sensory modalities.

92

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

45 40 35 30

*** * ***
Auditory-only Visual-only

Errors (%)

25 20 15 10 5 0 Expect audition Expect vision Low auditory Low visual load load Experiment 6 Perceptual load manipulation

Experiment 5 Probability manipulation

* ***

p < .05 p < .001

Figure 4.2. Figure showing the percentages of auditory-only and visual-only errors in the Expect audition and Expect vision conditions (where the ratios of the stimuli were 60A:20V:20AV and 20A:60V:20AV, respectively; Experiment 5), and in the Low auditory load and Low visual load conditions (Experiment 6), in Sinnett et al.’s (2007) study.

As varying the focus of participants’ endogenous attention has been shown to modulate the Colavita effect in a version of the Colavita task using complex, meaningful stimuli (Sinnett et al., 2007), the aim of Experiment 4.1 was to investigate whether a similar modulation of the effect could be found using simple stimuli. Given the potential response biases and repetition effects associated with using probability manipulations, a purely top-down instructional manipulation of attention was used instead (cf. Egeth & Sager, 1977, Experiment 6). This approach contrasts with the relatively stimulus-driven effects attributable to changing perceptual load or the changing the probabilities with which the various stimuli were presented (used in most of the previous research). Participants’ attention was directed endogenously toward a particular sensory modality by simply instructing them to expect stimuli to

93

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

occur in that sensory modality2. This aspect of the design was modelled on the results of many recent studies that have demonstrated significant modulations in behavioural performance and/or neural activation using purely instructional manipulations of participants’ attention (e.g., Büchel, Josephs, Rees, Turner, Frith, & Friston, 1998; Jäncke, Mirzazadeb, & Shah, 1999; O’Craven, Rosen, Kwong, Treisman, & Savoy, 1997; Shomstein & Yantis, 2004; Tse, 2005). The primary goal of Experiment 4.1 was therefore to investigate the influence of the modality toward which participants’ attention is directed endogenously on the magnitude of the Colavita effect by varying which modality participants were instructed to expect stimuli to be presented in, and observing the effect that this had on the pattern of errors that participants made on the bimodal target trials. The design was identical to that used in Experiment 2.1 with the exception that participants were presented with both an Expect audition (a set of four blocks of trials) and an Expect vision condition (a set of four blocks of trials) in which they were instructed to expect targets in either the auditory or visual modalities (in the Expect audition and Expect vision conditions, respectively). If this manipulation of attention is effective, then participants should respond more rapidly to targets in the attended modality than to the same targets when their attention was directed toward the other modality. Furthermore, if attention directed endogenously toward a sensory modality does indeed influence the Colavita effect (as claimed by Colavita & Weisberg, 1979; Egeth & Sager, 1977; Posner et al., 1976), one should expect to observe a larger Colavita effect in the Expect vision condition and a reversed (or at the very least, attenuated) Colavita effect in the Expect audition condition.

2

Note that the effect that this would have is similar to manipulations of stimulus probability (in that participants are likely to expect stimuli occurring in the most frequent modality), except that the frequencies of the stimuli and responses would not change across conditions.

94

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

4.1

EXPERIMENT 4.1

4.1.1 Methods Participants. 16 naïve participants (mean age of 26 years, age range from 1932 years; 3 males and 13 females) took part in Experiment 4.1. All except two of the participants were right-handed by self-report. The experimental session lasted for approximately 30 minutes. Apparatus, materials, design, and procedure. These were exactly the same as in Experiment 2.1 with the exceptions that the ISI was now 1500ms, and participants were presented with a set of four Expect audition blocks, and a set of four Expect vision blocks, with the order of the blocks counterbalanced across participants. In the Expect audition blocks, the participants were verbally instructed before each block of trials to direct their attention toward the auditory modality by expecting an auditory stimulus on every trial. The same procedure was adopted for the Expect vision blocks (except that participants were now instructed to direct their attention toward the visual modality and to expect visual targets instead). The participants were frequently encouraged and reminded to expect stimuli in the appropriate sensory modalities before the start of each block of trials.

4.1.2 Results Error data. The participants failed to respond on 0.3% of the trials overall, and these trials were not included in the data analyses. The RT and error data from Experiment 4.1 are shown in Table 4.1. An ANOVA performed on the data from the bimodal trials in which the participants failed to respond correctly with the factors of 95

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Response (Auditory-only or Visual-only) and Expected Modality (Audition or Vision), revealed no significant main effect of Response [F < 1, n.s.], nor any effect of Expected Modality [F(1, 15) = 1.58, p = .228]. Importantly, however, the analysis of the bimodal error data revealed a significant interaction between Response and Expected Modality [F(1, 15) = 13.78, p = .002], attributable to there being a significant Colavita visual dominance effect in the Expect vision condition (13.2% visual-only responses vs. 3.6% auditory-only responses; t(15) = 3.83, p = .002), and a significant reverse Colavita effect in the Expect audition condition (6.1% visual-only responses vs. 16.4% auditory-only responses; t(15) = 2.94, p = .010; i.e., auditory dominance was observed; see Figure 4.3). These results therefore demonstrate that the magnitude of the Colavita visual dominance effect (using simple stimuli) can be influenced by the particular sensory modality that participants are instructed to expect or attend to. An ANOVA performed on the error data with the factors of Target Stimulus (Auditory, Bimodal, or Visual) and Expected Modality (Audition or Vision) revealed a significant main effect of Target Stimulus [F(1.12, 16.83) = 7.36, p = .013], with participants making significantly more errors on the bimodal trials (20.5% errors) than on the unimodal auditory (7.5% errors; t(15) = 2.81, p = .013) or unimodal visual trials (8.0% errors; t(15) = 2.73, p = .016), but no more errors on unimodal visual trials than on the unimodal auditory trials (t(15) = 0.33, p = .746). None of the other terms reached significance: for Expected Modality [F < 1, n.s.], or for the interaction between the two terms [F(1.12, 16.78) = 1.14, p = .309].

96

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

**
24 22 20 18 16 Auditory-only Visual-only

*

**

Errors (%)

14 12 10 8 6 4 2 0 Expect audition Expect vision

* **

p < .05 p < .01

Figure 4.3. Figure showing the percentages of auditory-only and visual-only errors made by participants in the Expect audition and Expect vision conditions in Experiment 4.1. The error bars indicate the standard errors of the means.

RT data. The RT data from those trials in which the participants responded correctly were analysed in an ANOVA with the factors of Target Modality (Auditory or Visual), Target Type (Unimodal or Bimodal), and Expected Modality (Audition or Vision). The analysis revealed a significant main effect of Target Type [F(1, 15) = 91.95, p < .001], attributable to participants responding significantly more rapidly on unimodal (378ms) than on bimodal target trials (463ms). Although participants responded somewhat more rapidly to visual targets than to auditory targets overall (412ms vs. 430ms), the main effect of Target Modality just failed to reach statistical significance [F(1, 15) = 4.33, p = .060]. However, the interaction between Target Modality and Target Type was significant [F(1, 15) = 10.28, p = .006]. This term reflects the fact that participants responded significantly more rapidly to visual than to auditory targets on the bimodal trials (443ms vs. 484ms; t(15) = 2.78, p = .014),

97

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

whereas this difference failed to reach significance on the unimodal trials (381ms vs. 377ms; t(11) = .60, p = .555).
Table 4.1. Mean reaction auditory, unimodal visual, Expect vision conditions auditory, unimodal visual, shown in parentheses. times (RTs; ms) for correct responses to unimodal and bimodal target stimuli in the Expect audition and of Experiment 4.1. Mean error rates for unimodal and bimodal target stimuli target. Standard errors are

Expected modality RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses Errors (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses

Expect audition

Expect vision

351 (23) 411 (26)

403 (17) 351 (19)

451 (32) 470 (29)

517 (24) 415 (30)

5.5 (1.3) 9.2 (1.7)

6.3 (1.1) 6.0 (1.0)

16.4 (4.8) 6.1 (1.9)

3.6 (1.1) 13.2 (3.1)

There was also a significant interaction between Target Modality and Expected Modality [F(1, 15) = 25.74, p < .001], attributable to participants responding more rapidly to the visual targets than to the auditory targets in the Expect vision condition (mean difference = 77ms; t(15) = 4.17, p = .001), whereas the reverse was true in the Expect audition condition (mean difference = 40ms; t(15) = 4.27, p = .001; see Table 4.1). The participants responded more rapidly to the visual targets in the Expect vision condition than in the Expect auditory condition (mean difference = 58ms; t(15) = 3.64, p = .004), whereas the reverse was true for auditory targets (mean difference = 59ms; t(15) = 3.46, p = .002). None of the other terms were significant: for Expected Modality, Expected Modality × Target Type, or for Expected Modality × Target Modality × Target Type, for all terms [F < 1, n.s.].

98

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

4.1.3 Discussion The results of Experiment 4.1 revealed a significant Colavita visual dominance effect in the Expect vision condition (that is, participants made significantly more visual-only than auditory-only responses in the Expect vision condition; 13.2% vs. 3.6%, respectively), whereas a significant reverse Colavita effect (i.e., auditory dominance) was observed in the Expect audition condition (16.4% auditory-only vs. 6.1% visual-only responses, respectively). These results therefore demonstrate that the Colavita effect can be modulated by directing a participant’s attention endogenously toward a particular sensory modality, and that the effect can be eliminated (and even reversed) when participants are asked to direct their attention endogenously toward the auditory modality instead. While Sinnett et al. (2007) suggested that directing attention endogenously toward the auditory modality can only attenuate the magnitude of the Colavita effect (at least in their manipulation of perceptual load in which response biases presumably did not contribute to their results), the results of Experiment 4.1 go further. They provide the first empirical evidence that shows that endogenously attending to the auditory modality can reverse the Colavita effect (and give rise to a significant effect of auditory dominance over vision). One reason for this difference might be related to the possibility that the more top-down and instructional nature of the manipulation of participants’ expectations in Experiment 4.1 may have provided a more effective means of manipulating participants’ attention endogenously than the more bottom-up stimulus-driven stimulus probability, or perceptual load, manipulations used by Sinnett and his colleagues.

99

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Instructing participants to expect stimuli in a particular sensory modality improved their performance in reacting to targets presented in that modality; participants responded more rapidly to targets presented in the expected sensory modality than to the same targets when their attention was directed to the other sensory modality (both in the case of unimodal and bimodal targets). In order to be certain that the manipulation of the modality expectation used in Experiment 4.1 really did manipulate participants’ attention endogenously as claimed, it is necessary to rule out other explanations such as criterion shifting, and response bias effects, which could perhaps also account for these effects (see Spence & Driver, 1997b). That is, it is possible that the manipulation of modality expectation may have caused a criterion shift rather than having changed the modality that participants were attending to (Mulligan & Shaw, 1981; see Spence & Driver, 1997b, for a review). According to the criterion shifting account, the participants may simply have lowered their criteria (i.e., become more liberal) in responding to targets in the sensory modality that they had verbally been instructed to expect targets in. Along the same lines, it could also be argued that participants may simply have been biased to press the response key corresponding to the expected modality. If this were to have been the case, one would have expected to see participants making more inappropriate responses3 to targets presented in the expected modality. This, however, was not the case (see Table 4.1). Indeed, the error data revealed no significant main effects or interactions (i.e., participants did not make significantly more false alarms corresponding to responses to targets presented in the expected modality), suggesting that the false alarm rates were not affected by the instruction to attend to one modality

3

Remember, the percentage of unimodal errors is indicative of the percentage of trials in which participants made inappropriate responses (i.e., responding to a stimulus that was not presented on a given trial; e.g., on unimodal auditory trials, this would involve pressing the visual response key), as discussed in Chapter 2 (Section 2.1.2).

100

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

versus the other. The fact that the false alarm rates were unaffected by which modality participants had been instructed to expect shows that they did not simply lower their criteria for responding to the expected modality, nor were they more biased to press the response key corresponding to the expected modality. Finally, it is worth noting that purely instructional manipulations have been used by a number of other researchers in recent years (Büchel et al., 1998; Jäncke et al., 1999; O’Craven et al., 1997; Shomstein & Yantis, 2004; Tse, 2005), and in none of these studies were response biases reported, nor were the effects obtained explainable by response bias4. Thus, it is unlikely that the results of Experiment 4.1 are due to response biases or criterion shifting effects.

4.1.4 Between-participants analysis of Experiments 2.1 and 4.1 In order to determine what the effects of attending to the auditory or visual modalities on the magnitude of the Colavita effect were, two between-participants comparisons were conducted between the Colavita effect observed in Experiment 2.1 and the Colavita effects observed in the Expect audition and Expect vision conditions of Experiment 4.1. The only difference between Experiments 2.1 and 4.1 was that participants were instructed to attend to the auditory or visual modalities (Expect audition and Expect vision conditions) in Experiment 4.1, whereas no such instructions to attend to a particular modality were given in Experiment 2.1. Therefore, any differences between Experiment 2.1 and the Expect audition and

For example, O’Craven et al. (1997) used functional magnetic resonance imaging (fMRI) to examine the attentional modulation of neural activity in MT-MST (a brain area involved in the processing of visual motion). Using a visual stimulus containing both moving and stationary dots, they found significantly more MT–MST activation when subjects were instructed to attend to the moving dots than when they attended to the stationary dots (despite the fact that the visual stimulus was identical in both conditions). This pattern of brain activation corresponding to the type of stimulus participants attended to cannot be explainable by response biases.

4

101

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Expect vision conditions reflect the effects of attending to the auditory or visual modalities on the magnitude of the Colavita effect (versus not being instructed to attend to either modality). The bimodal error data from the Expect vision condition in Experiment 4.1 were compared with the data from Experiment 2.1 in a between-participants ANOVA with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Expected Modality (Expect vision or None). The analysis revealed a significant main effect of Response [F(1, 28) = 19.71, p < .001], with the participants making significantly more visual-only (12.6% of all bimodal trials) than auditory-only responses (4.0% of all bimodal trials), overall. The main effect of Expected Modality was, however, not significant (i.e., there was no overall difference in the percentage of bimodal errors made between participants in Experiments 2.1 and 4.1). There was no significant interaction between Response and Expected Modality, for the two terms [F < 1, n.s.]. Thus, the magnitude of the Colavita effect observed in the Expect vision condition in Experiment 4.1 was not significantly different from that observed in Experiment 2.1. A similar analysis was carried out between the bimodal error data from the Expect audition condition in Experiment 4.1 and the data from Experiment 2.1, with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Expected Modality (Expect audition or None). There were no significant main effects of Expected Modality or Response, for both terms [F < 1, n.s.]. However, there was a significant interaction between Response and Expected Modality [F(1, 28) = 14.70, p = .001], attributable to the Colavita effect being significant in Experiment 2.1 where participants were not attending to a particular modality (12.1% visual-only vs. 4.4% auditory-only errors; t(13) = 2.53; p

102

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

= .025), and being reversed in the Expect audition condition (6.1% visual-only responses vs. 16.4% auditory-only responses; t(15) = 2.94, p = .010). Thus, the Colavita effect was significantly modulated by instructing participants to attend to the auditory modality. The finding that the magnitude of the Colavita effect was not increased by having participants attend to the visual modality (as in the Expect vision condition of Experiment 4.1) versus when they were not instructed to attend to a particular modality (as in Experiment 2.1), could be taken to suggest that participants in Experiment 2.1 may already have been attending to the visual modality (even though they had not been instructed to). That is, these results could be taken to support the argument that the Colavita effect may be contributed to, or caused by, participants having a tendency5 to direct their attention endogenously toward the visual modality (e.g., Colavita & Weisberg, 1979; Egeth & Sager, 1977). The fact that participants tended to respond faster to visual than to auditory stimuli6 in the majority of experiments reported in this thesis, the support in the literature for the notion that people have a general bias to endogenously attend toward the visual modality (Hohnsbein, Falkenstein, Hoormann, & Blanke, 1991; Klein, 1977; Posner et al., 1976; even under conditions of divided attention, Spence et al., 2001b; Zampini,

However, it should be noted that the results (no significant difference in the magnitude of the Colavita effect between Experiments 2.1 and 4.1) are equally consistent with the possibility that another factor (i.e., not endogenous attention toward vision) may have caused the Colavita effect observed in Experiment 2.1. 6 RTs can provide a rough indication of the modality toward which participants are attending (as discussed in Chapter 2, Section 2.1.3). Note that this argument (that participants attending to vision would respond more rapidly to the visual than to the auditory stimulus) is equally consistent with the suggestion that the faster responses to the visual stimulus are due simply to the visual stimulus being much more intense than the auditory stimulus. However, it is important to remember that the stimuli were presented clearly above threshold levels, and that interactions occurring between stimuli presented well above threshold levels are not affected to a great extent by the variation in the relative intensities of the stimuli (see Spence et al., 2001b). Therefore, it is plausible to suggest that the participants’ relative response latencies to the auditory and visual stimuli are caused by them attending to vision rather than being caused by the relative intensities of the stimuli.

5

103

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Shore, & Spence, 2005b), and the finding that the Colavita effect was reversed by instructing participants to attend to the auditory modality (i.e., directing their attention away from the visual modality), provide further support for this notion that participants do endogenously attend toward vision during the performance of the Colavita task, and this tendency to endogenously attend toward vision contributes to the Colavita effect.

4.1.5 Exogenous attention and the Colavita effect While Posner et al. (1976) claimed that the bias to attend toward the visual modality was endogenously (voluntarily) mediated, Spence et al. (2001a) suggested that attentional biases toward the visual modality might instead be exogenously (i.e., reflexively) mediated. That is, visual stimuli may capture a person’s attention exogenously more effectively than stimuli in other sensory modalities (e.g., Turatto, Benso, Galfano, Gamberini, & Umiltà, 2002; see also Hamlin, 1895; Smith, 1933, on this point) which would result in a shift of attention toward the visual modality (e.g., Rodway, 2005; Spence et al., 2001a). Along these lines, it could be argued that the Colavita effect is caused by visual stimuli having a greater ability to exogenously (or automatically) attract attention toward themselves on bimodal trials than auditory stimuli do. Experiment 4.2 was therefore designed to investigate whether the Colavita effect could be modulated by the sensory modality toward which participants attention is directed exogenously. The effects of directing attention exogenously toward different sensory modalities have typically been studied using crossmodal exogenous cuing paradigms in which two stimuli are presented one after the other in close succession (i.e., the target stimulus is preceded by the exogenous cue). In such paradigms, the participants 104

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

have to respond to the target stimulus but not to the cue (in fact, participants are often told to ignore the exogenous cue because it is irrelevant to the task). It is generally found that the modality of the exogenous cue affects the processing of the target stimulus in some way; in particular, the modality of the exogenous cue usually facilitates the processing of an ipsimodal (i.e., same modality) target stimulus, but slows the processing of a crossmodal (i.e., different modality) target stimulus (e.g., Posner, 1978; Turatto et al., 2002; Turatto, Galfano, Bridgeman, & Umiltà, 2004). This occurs despite the fact that the exogenous cues provide no information concerning the likely modality of the target stimuli. Using a crossmodal exogenous cuing paradigm, Turatto et al. (2002) reported an experiment in which they found an asymmetry in the effectiveness of auditory and visual exogenous cues in modulating responses to subsequently-presented target stimuli; visual cues were found to be more effective at exogenously drawing participants’ attention away from the auditory target stimuli than auditory cues were (at exogenously drawing attention away from visual target stimuli). In their study, Turatto et al. (2002, Experiment 4) had participants allocate their attention endogenously toward one sensory modality (either audition or vision; by fixing the modality of the target stimuli within each block of trials), and varied the modality of the exogenous cue. The results revealed that when attention was endogenously allocated to the visual modality, the detection of a visual target was not slowed by the presentation of an auditory exogenous cue, which Turatto et al. interpreted as showing that the auditory exogenous cues were not effective at shifting attention away from vision. In contrast, when attention was endogenously allocated to audition, the detection of an auditory target stimulus was slowed by the prior presentation of a visual exogenous cue. Hence, Turatto et al. argued that visual cues are simply more

105

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

effective at exogenously drawing participants’ attention away from the auditory modality than vice versa, and further suggested that there may be a higher priority interrupt for visual inputs, which causes an obligatory processing of visual stimuli. Experiment 4.2 was performed to investigate whether the Colavita effect can be modulated by the sensory modality toward which participants’ attention is directed exogenously, and also to determine whether the visual components of bimodal stimuli are more effective at capturing participants’ attention exogenously than the auditory components are (as suggested by the results of Turatto et al.’s, 2002, study). The design of Experiment 4.2 was therefore similar to that used in Experiment 2.1 with the exception that an exogenous cue (either auditory or visual) always preceded the presentation of the target stimulus. If the exogenous cues are effective at directing attention toward a sensory modality, then one would expect to observe faster RTs for auditory stimuli (whether they are presented as a unimodal target, or as part of a bimodal target) when preceded by an ipsimodal auditory cue than when preceded by a crossmodal visual cue, and vice versa for visual stimuli. If the modality toward which participants’ attention is exogenously directed contributes to the Colavita visual dominance effect, then the Colavita effect should be larger on those bimodal trials in which attention is exogenously directed toward the visual modality, and attenuated (or reversed) when attention is exogenously directed toward the auditory modality.

4.2

EXPERIMENT 4.2

4.2.1 Methods Participants. 20 naïve participants (mean age of 24 years, age range from 19 34 years; 5 males and 15 females) took part in Experiment 4.2. All except one of the

106

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

participants were right-handed by self-report. The experimental session lasted for approximately 30 minutes. Apparatus and materials. These were exactly the same as in Experiment 2.1 with the exception that exogenous auditory and visual cues were presented on each trial prior to the presentation of the target. The exogenous auditory and visual cues7 consisted of the presentation of a 2000Hz pure tone for 50ms, or the illumination of a red LED for 50ms, respectively, and were presented from exactly the same position as the target stimuli (cf. Turatto et al., 2002, 2004; Rodway, 2005, who used similar methodologies). The participants were informed that the targets on each trial would be preceded by an irrelevant exogenous cue which they should ignore. Design. The participants were presented with 8 blocks of 100 trials, each consisting of 40 visual trials, 40 auditory trials, and 20 bimodal trials. A target stimulus could be preceded by an exogenous visual or exogenous auditory cue. The two types of cue were presented equally often within a block of trials, and equally often preceding each type of target stimulus. Procedure. The procedure was exactly the same as in Experiment 2.1, with the sole exception that on each trial, an exogenous visual or auditory cue was presented 200ms before the onset of the target stimulus.

Remember, the target auditory and visual stimuli consisted of the presentation of a 4000Hz tone, and the illumination of a yellow LED, respectively.

7

107

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

4.2.2 Results
16 14 12 10 8 6 4 2 0 Auditory exogenous cue Visual exogenous cue Auditory-only Visual-only

* *

Errors (%)

*

p < .05

Figure 4.4. Figure showing the percentages of auditory-only and visual-only errors when the bimodal targets were preceded by exogenous visual and exogenous auditory cues in Experiment 4.2. The error bars indicate the standard errors of the means.

Error data. The participants failed to respond on 0.9% of the trials, and these trials were not included in the data analyses. The RT and error data are shown in Table 4.2. The data from the bimodal trials in which the participants failed to respond to one of the two stimuli were analysed using an ANOVA with the factors of Response (Auditory-only or Visual-only) and Cue Modality (Auditory or Visual). The analysis revealed a significant main effect of Response [F(1, 19) = 5.85, p = .026], with the participants making significantly more visual-only than auditory-only responses (9.2% vs. 3.5% of all bimodal trials, respectively) overall. The analysis also revealed a significant interaction between Response and Cue Modality [F(1, 19) = 4.65, p = .044], attributable to the fact that the Colavita effect was significantly larger when an exogenous visual cue preceded the bimodal target stimulus than when an exogenous auditory cue preceded it (7.2% vs. 4.2%; t(19) = 2.16, p = .044; see Figure 4.4). The main effect of the Cue Modality was, however, not significant [F < 1, n.s.]. 108

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Table 4.2. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli which were preceded by auditory or visual exogenous cues in Experiment 4.2. Mean error rates for unimodal auditory, unimodal visual, and bimodal target stimuli target. Standard errors are shown in parentheses. Exogenous cue modality RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses Errors (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses Auditory Visual

522 (34) 562 (29)

558 (30) 518 (26)

652 (32) 640 (31)

685 (32) 651 (35)

3.0 (0.6) 9.3 (1.6)

10.7 (3.7) 6.1 (1.4)

4.0 (0.7) 8.2 (2.2)

3.1 (0.7) 10.3 (3.1)

An ANOVA performed on the error data for the unimodal and bimodal trials with the factors of Target Stimulus (Auditory, Bimodal, or Visual) and Cue Modality (Auditory or Visual) revealed a significant interaction between the two factors [F(1.29, 24.41) = 4.95, p = .028]. The participants responded significantly more accurately to unimodal visual targets preceded by an ipsimodal (i.e., same modality) than by a crossmodal (i.e., different modality) exogenous cue (6.0% vs. 9.2% errors; t(19) = 4.11, p = .001), whereas this effect just failed to reach significance for the unimodal auditory targets (3.0% vs. 10.7% errors; t(19) = 2.08, p = .052). None of the other terms in this analysis reached significance: for the main effects of Cue Modality [F(1, 19) = 2.19, p = .155], and Target Stimulus [F(1.22, 23.24) = 2.68, p = .109]. RT data. The RT data from those trials in which the participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), Target Type (Unimodal or Bimodal), and Cue Modality

109

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

(Auditory or Visual). The analysis revealed a significant main effect of Target Type [F(1, 19) = 102.33, p < .001], attributable to the participants responding significantly more rapidly on the unimodal (540ms) than on the bimodal target trials (657ms). The analysis also revealed a significant interaction between Target Modality and Cue Modality [F(1, 19) = 28.28, p < .001]. The participants responded significantly more rapidly to auditory targets that had been preceded by an ipsimodal (auditory) cue than by a crossmodal (visual) cue (587ms vs. 621ms; t(19) = 3.12, p = .006). This effect was borderline-significant effect for visual targets (584ms vs. 601ms; t(19) = 2.01, p = .059). This asymmetry in the magnitude of the effect of Cue Modality on the RT data was significant (t(19) = 5.32, p < .001). Finally, there was a significant interaction between Target Type, Target Modality, and Cue Modality [F(1, 19) = 14.81, p = .001]. This interaction was attributable to the fact that the participants responded more rapidly to targets which had been preceded by an ipsimodal cue than by a crossmodal cue, for unimodal auditory, bimodal auditory, and unimodal visual targets (mean difference = 36ms, 33ms, and 44ms, respectively; t(19) = 3.21, p = .005; t(19) = 2.42, p = .026; t(19) = 4.27, p < .001), whereas a non-significant trend in the reverse direction was observed for bimodal visual targets (11ms; t(19) = .79, p = .440). On the unimodal trials, the participants responded more rapidly when a visual cue had been presented than when an auditory cue had been presented, whereas the reverse was true for performance on the bimodal trials. This difference was borderline-significant (mean differences = 4ms vs. 22ms; t(19)= 2.20, p = .058), resulting in a borderline-significant interaction between Target Type and Cue Modality [F(1, 19) = 4.11, p = .057]. None of the other terms in the analysis reached significance: for the main effects of Target Modality

110

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

[F(1, 19) = 3.12, p = .093], Cue Modality [F(1, 19) = 1.05, p = .318], or for the interaction between Target Modality × Target Type [F(1, 19) = 3.17, p = .091].

4.2.3 Discussion The results of Experiment 4.2, once again, revealed a significant Colavita visual dominance effect, with the participants making significantly more visual-only than auditory-only responses overall (9.2% vs. 3.5% of all bimodal trials). Furthermore, the magnitude of the Colavita effect was significantly attenuated (but not reversed; cf. Experiment 4.1) for the bimodal target stimuli that had been preceded by auditory exogenous cues than for those stimuli that had been preceded by visual exogenous cues (4.2% vs. 7.2%). One reason for why directing a participant’s attention exogenously toward audition had a smaller effect on the Colavita effect (i.e., it attenuated the Colavita effect) than directing a participant’s attention endogenously toward audition (i.e., it reversed the Colavita effect) may have been because visual stimuli are more effective than auditory stimuli at capturing a participant’s attention exogenously (as discussed in the following paragraph). On a certain proportion of the bimodal trials, the visual component of the bimodal stimulus may simply have captured a participant’s attention exogenously regardless of the exogenous cue that had preceded it (i.e., even on trials in which an auditory exogenous cue had exogenously directed a participant’s attention toward audition, the visual component of the bimodal target stimulus may have exogenously captured the participant’s attention back toward vision). The finding that participants responded faster to the visual than to the auditory stimulus regardless of the modality of the preceding exogenous cue supports this argument.

111

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

The participants in Experiment 4.2 responded significantly more slowly to unimodal auditory targets that had been preceded by a crossmodal (visual) exogenous cue than by an ipsimodal (auditory) exogenous cue. In contrast, this trend (responding more slowly to targets preceded by crossmodal than ipsilesional cues) was significantly smaller for the visual targets. These findings support the argument that the exogenous cues were effective in directing participants’ attention toward the particular modalities that they were presented in, and are also consistent with those reported by Turatto et al. (2002; see also Posner, 1978), who showed that auditory exogenous cues did not slow simple detection responses to visual target stimuli as much as visual exogenous cues slowed simple detection responses to auditory targets. It should, however, be noted that the participants in Experiment 4.2 responded significantly more accurately to visual targets which had been preceded by an ipsimodal than by a crossmodal exogenous cue (that is, they made more auditory false-alarms to targets that followed an auditory exogenous cue than a visual exogenous cue). In contrast, this effect just failed to reach significance for the unimodal auditory targets (that is, they did not make significantly more visual falsealarms to targets which followed a visual exogenous cue than to targets followed by an auditory exogenous cue). This finding suggests that the attenuation of the Colavita effect observed on those bimodal trials which had been preceded by an auditory exogenous cue may have been partly due to participants being more biased to pressing the auditory response key when an auditory exogenous cue had been presented.

4.2.4 Between-participants analysis of Experiments 2.1 and 4.2 To determine the effects of the presentation of auditory and visual exogenous cues on the magnitude of the Colavita effect, two between-participants comparisons 112

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

were conducted between the Colavita effect observed in Experiment 2.1 and those observed in the Auditory and Visual exogenous cue conditions of Experiment 4.2. As the only difference between Experiments 2.1 and 4.2 was that participants were presented with auditory and visual exogenous cues in Experiment 4.2 (whereas no cues were presented in Experiment 2.1), any differences in the magnitude of the Colavita effect between Experiment 2.1 and the Auditory and Visual exogenous cue conditions in Experiment 4.2 reflect the effects of the exogenous cues. The bimodal error data from the Visual exogenous cue condition in Experiment 4.2 were compared with the data from Experiment 2.1 in a betweenparticipants ANOVA with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Cue Modality (Visual or None). The analysis revealed a significant main effect of Response [F(1, 32) = 12.67, p < .001], with the participants making significantly more visual-only (11.1% of all bimodal trials) than auditory-only responses (3.7% of all bimodal trials), overall. None of the other terms reached significance: for the main effect of Expected Modality, or the interaction between Response and Expected Modality, for both terms [F < 1, n.s.]. Thus, the magnitude of the Colavita effect observed in the Visual exogenous cue condition was not significantly different from that observed in Experiment 2.1 in which participants were not presented with exogenous cues. A similar analysis was carried out between the bimodal error data from the Auditory exogenous cue condition in Experiment 4.1 and the data from Experiment 2.1, with the within-participants factor of Response (Auditory-only or Visual-only), and the between-participants factor of Cue Modality (Auditory or None). Once again, the analysis revealed a significant main effect of Response [F(1, 32) = 11.18, p = .002], with the participants making significantly more visual-only (10.1% of all

113

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

bimodal trials) than auditory-only responses (4.2% of all bimodal trials), overall. However, none of the other terms reached significance: for the main effect of Expected Modality [F(1, 32) = 1.15, p = .291], or the interaction between Response and Expected Modality [F < 1, n.s.]. In sum, the magnitude of the Colavita effect (as observed in Experiment 2.1, in which no exogenous cues were presented) was not significantly decreased by exogenously directing participants’ attention to the auditory modality (as in the Auditory exogenous cue condition) nor significantly increased by exogenously directing participants’ attention to the visual modality (Visual exogenous cue condition). The fact that the magnitude of the Colavita effect in the Visual exogenous cue condition was not significantly larger that observed in Experiment 2.1 could be taken to suggest that the Colavita effect observed in Experiment 2.1 may already have been due, at last in part, to participants’ attention being captured by the visual stimulus on bimodal trials8. That is, the visual component of the bimodal stimulus may have exogenously captured participants’ attention (away from the auditory stimulus) on the bimodal trials in Experiment 2.1. The finding that the Colavita effect was not significantly attenuated by the presentation of an auditory exogenous cue (compared to when no cues were presented) could be taken to support the notion that auditory stimuli are not as effective as visual stimuli are in exogenously drawing attention toward themselves (as also suggested by the RT data from Experiment 4.2; see also Turatto et al., 2002).

It should be noted, once again, that the results (no significant difference in the magnitude of the Colavita effect between Experiments 2.1 and 4.2) are equally consistent with the possibility that another factor (i.e., not exogenous attention toward vision) may have caused the Colavita effect observed in Experiment 2.1.

8

114

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

To summarize the findings of the two experiments reported so far in this chapter, the results of Experiment 4.1 support the notion that the Colavita effect could be caused, or contributed to, by participants having the tendency to endogenously attend toward the visual modality. The results of Experiment 4.2 suggest that visual exogenous cues may be more able to attract attention than auditory exogenous cues (cf. Turatto et al., 2002), and this may also be a factor that contributes to the emergence of the Colavita effect. However, it appears that exogenous auditory cues (presented just before the bimodal targets) are unable to significantly attenuate the Colavita effect (at least compared to conditions in which no exogenous cues are presented). As directing attention both endogenously and exogenously toward the visual modality appears to contribute to the Colavita effect, Experiment 4.3 was conducted in order to investigate the relative contributions of attention directed endogenously and attention directed exogenously to the Colavita effect. The design was similar to that used in Experiment 2.1 with the exception that the participants’ attention was manipulated both endogenously and exogenously (as in Experiments 4.1 and 4.2). Turatto et al. (2002) demonstrated that visual cues were able to exogenously draw participants’ attention away from the auditory modality (even when participants were endogenously attending to audition; as discussed in Section 4.1.5), whereas auditory cues were not effective at drawing attention away from vision. Indeed, the RT data from Experiment 4.2 supports the notion that visual stimuli are more effective at exogenously attracting attention than auditory stimuli. Thus, on the basis of these findings, one might predict that the magnitude of the Colavita effect would be unaffected by auditory exogenous cues when participants are endogenously attending to the visual modality. However, when participants are endogenously attending to the

115

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

auditory modality, the magnitude of the reverse Colavita effect (auditory dominance) may be attenuated by the presentation of visual exogenous cues.

4.3

EXPERIMENT 4.3

4.3.1 Methods Participants. 12 naïve participants (mean age of 23 years, age range from 19 29 years; 2 males and 10 females) took part in Experiment 4.3. All except one of the participants were right-handed by self-report. The experimental session lasted for approximately 35 minutes. Apparatus, materials, design, and procedure. These were exactly the same as in Experiment 4.1 with the exception that exogenous auditory and visual cues were presented on each trial 200ms prior to the presentation of the target (as in Experiment 4.2).

4.3.2 Results Error data. The participants failed to respond on 1.3% of the trials overall, and these trials were not included in the data analyses. The RT and error data are shown in Table 4.3. The data from the bimodal trials in which the participants failed to respond to one of the two stimuli were analysed using an ANOVA with the factors of Response (Auditory-only or Visual-only), Expected Modality (Audition or Vision), and Cue Modality (Auditory or Visual). The analysis revealed a significant main effect of Response [F(1, 11) = 10.76, p = .007], with the participants making significantly more visual-only than auditory-only responses (8.4% vs. 4.6% of all bimodal trials, respectively) overall. The analysis also revealed a significant 116

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

interaction between the Response and Expected Modality [F(1, 11) = 9.37, p = .011], attributable to the fact that there were significantly more visual-only than auditoryonly errors in the Expect vision condition (10.3% visual-only vs. 2.4% auditory-only errors; t(11) = 3.76, p = .003), whereas the reverse trend failed to reach significance in the Expect audition condition (6.6% visual-only vs. 6.8% auditory-only errors; t(11) = .12, p = .904). There was also a significant interaction between the Response and Cue Modality [F(1, 11) = 7.27, p = .021], attributable to participants making significantly more visual-only than auditory-only errors when a visual exogenous cue preceded the bimodal target (11.9% vs. 3.0% visual-only and auditory-only errors; t(11) = 3.08, p = .010), whereas the reverse trend failed to reach significance when an auditory exogenous cue preceded the target (4.9% vs. 6.1% visual-only and auditory-only errors; t(11) = 1.04, p = .332). Finally, there was a significant interaction between Response, Expected Modality, and Cue Modality [F(1, 11) = 4.93, p = .048], which was attributable to the Colavita visual dominance effect being significant in the Expect vision-Auditory exogenous cue and Expect vision-Visual exogenous cue conditions, whereas this trend failed to reach significance in the Expect audition-Visual exogenous cue condition (visual dominance effect = 5.4%, 10.2%, and 7.5%, respectively; t(11) = 3.88, p = .003; t(11) = 3.16, p = .009; t(11) = 1.78, p = .105; see Figure 4.5). In contrast, the reverse Colavita effect (i.e., more auditory-only than visual-only errors auditory dominance) was present in the Expect audition-Auditory exogenous cue (auditory dominance = 7.8%; t(11) = 3.78, p = .003). None of the other terms reached significance: for Expected Modality [F <1, n.s.], Cue Modality [F(1, 11) = 3.33, p = .095], or for the interaction between Expected Modality × Cue Modality [F(1, 11) = 1.01, p = .337].

117

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

*
18 16 14 12

*

**

**

Errors (%)

10 8 6 4 2 0 Auditory exogenous cue Visual exogenous cue Auditory exogenous cue Visual exogenous cue

Auditory-only Visual-only

Expect audition

Expect vision

* **

p < .05 p < .01

Figure 4.5. Figure showing the percentages of auditory-only and visual-only errors when the bimodal targets were preceded by auditory and visual exogenous cues in the Expect audition and Expect vision conditions in Experiment 4.3. The error bars indicate the standard errors of the means.

An ANOVA performed on the error data for the unimodal and bimodal trials with the factors of Target Stimulus (Auditory, Bimodal, or Visual), Expected Modality (Audition or Vision), and Cue Modality (Auditory or Visual) revealed no significant main effects or interactions: for the main effects of Target Stimulus [F(1.23, 24.41) = 2.42, p = .139], Expected Modality [F < 1, n.s.], or Cue Modality [F < 1, n.s.], or for the interactions between Target Stimulus × Expected Modality [F(2, 22) = 1.47, p = .235], Target Stimulus × Cue Modality [F(2, 22) = 2.45, p = .109], Expected Modality × Cue Modality [F(1, 11) = 1.73, p = .216], or Target Stimulus × Expected Modality × Cue Modality [F(2, 22) = 2.28, p = .126]. RT data. The RT data from those trials in which the participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), Target Type (Unimodal or Bimodal), Expected Modality

118

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

(Audition or Vision), and Cue Modality (Auditory or Visual). The analysis revealed a significant main effect of Target Type [F(1, 11) = 85.02, p < .001], attributable to participants responding more rapidly on the unimodal (494ms) than on the bimodal target trials (591ms). The analysis also revealed a significant main effect of Target Modality [F(1, 11) = 7.62, p = .019], with participants responding significantly more rapidly to visual (523ms) than to auditory targets (561ms), overall. Participants responded significantly more rapidly when they were expecting auditory stimuli (526ms) than visual stimuli (558ms), resulting in a significant main effect of Expected Modality [F(1, 11) = 5.34, p = .041].
Table 4.3. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli which were preceded by auditory or visual exogenous cues in the Expect audition and Expect vision conditions in Experiment 4.3. Mean error rates for unimodal auditory, unimodal visual, and bimodal target stimuli target. Standard errors are shown in parentheses. Expected modality Exogenous cue RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses Errors (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses Expect audition Auditory Visual Expect vision Auditory Visual

458 (34) 504 (35)

473 (30) 477 (30)

530 (41) 487 (40)

561 (38) 462 (42)

557 (40) 586 (47)

568 (38) 586 (47)

665 (38) 538 (43)

679 (41) 546 (58)

4.1 (1.5) 11.4 (2.2)

6.8 (2.1) 9.1 (1.4)

5.4 (1.1) 10 (1.9)

6.7 (1.6) 7.1 (1.8)

10.0 (2.6) 2.1 (1.1)

3.6 (1.3) 11.1 (4.2)

2.3 (0.8) 7.8 (1.7)

2.5 (0.8) 12.8 (3.4)

Finally, there was a significant interaction between Expected Modality and Target Modality [F(1, 11) = 21.55, p = .001], due to the fact that participants responded significantly more rapidly to visual than to auditory targets in the Expect

119

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

vision condition (508ms vs. 609ms, respectively; t(11) = 4.36, p = .001), whereas the reverse pattern failed to reach significance in the Expect audition condition (538ms vs. 514ms, for visual and auditory targets, respectively; t(11) = 1.69, p = .120). The interaction between Cue Modality and Target Modality failed to reach significance [F(1, 11) = 2.35, p = .154]. None of the other terms reached significance: for the interactions between Target Modality × Target Type [F(1, 11) = 3.96, p = .072], Target Modality × Target Type × Expected Modality [F(1, 11) = 2.46, p = .145], and for Target Modality × Target Type × Cue Modality [F(1, 11) = 2.49, p = .143]. For all other terms [F < 1, n.s.].

800

700

*

**

**
Auditory targets Visual targets

RT (ms)

600

500

400

300 Auditory exogenous cue Visual exogenous cue Auditory exogenous cue Visual exogenous cue

Expect audition

Expect vision

* **

p < .05 p < .01

Figure 4.6. Figure showing the mean RTs to auditory and visual targets (collapsed across the unimodal and bimodal RT data) which were preceded by auditory and visual exogenous cues in the Expect audition and Expect vision conditions in Experiment 4.3. The error bars indicate the standard errors of the means.

4.3.3 Discussion In Experiment 4.3, the Colavita effect was significant in both of the Expect vision conditions (whether the bimodal targets were preceded by auditory or visual 120

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

exogenous cues; Colavita effect = 5.4% and 10.2%, respectively), and this trend failed to reach significance in the Expect audition-Visual exogenous cue condition (Colavita effect = 7.5%9). In contrast, auditory dominance was observed in the Expect audition condition in which the bimodal targets were preceded by auditory exogenous cues (auditory dominance = 7.8%; see Figure 4.5 for a summary of these results). This pattern of results can be explained in terms of visual stimuli having a greater capacity to exogenously attract attention to themselves than auditory stimuli (as discussed in Sections 4.1.5 and 4.2.4). The Colavita visual dominance effect which occurs when participants are attending to the visual modality should be unaffected by the presentation of auditory exogenous cues, because auditory stimuli are unable to exogenously attract attention away from vision when attention is endogenously directed toward vision. Indeed, this was the case in the results of Experiment 4.3; the Colavita effect emerged in the Expect vision-Auditory exogenous cue condition. In contrast, the auditory dominance which occurs when participants attend to the auditory modality should be reversed or attenuated (i.e., visual dominance should be observed, or at least the auditory dominance should be attenuated) when visual exogenous cues are presented, because vision is effective at exogenously capturing attention away from the auditory modality. Indeed, in the Expect audition-Visual exogenous cue condition, the visual exogenous cues contributed to the visual dominance observed (albeit, the non-significant trend toward visual dominance). It is interesting to note that the pattern of differences observed in the RT data are similar to those observed in the bimodal error data (compare Figures 4.5 and 4.6).

The reason why the Expect audition-Visual exogenous cue condition Colavita effect did not reach significance (even though the Colavita effect was numerically larger (7.5%) than the Colavita effect in the Expect vision-Auditory exogenous cue condition (5.4%)) was because of the large standard errors in the Expect audition-Visual exogenous cue condition (see Figure 4.5).

9

121

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

In particular, the Colavita visual dominance effect was observed in the same conditions as those in which the visual bimodal RTs were faster than the auditory bimodal RTs (i.e., in the Expect vision-Auditory exogenous cue and Expect visionVisual exogenous cue conditions), and auditory dominance was observed in the same condition as that in which auditory bimodal RTs were faster than visual bimodal RTs (i.e., the Expect audition-Auditory exogenous cue). This finding ties back to the argument made in Chapter 2 (Section 2.1.3) that the more rapid RTs observed in response to visual than auditory stimuli on bimodal trials may be a product of participants endogenously attending toward vision. This point will be discussed further in the General discussion of this chapter. The error data revealed no significant main effects or interactions, which suggests that responses biases (to press the key corresponding with the expected modality or the modality of the exogenous cue) did not contribute to the pattern of results obtained in Experiment 4.3.

4.3.4 Between-participants analysis of Experiments 4.1 and 4.3 In order to determine the ability of auditory and visual exogenous cues to attract participants’ attention when they were endogenously attending to the auditory or visual modalities, four between-participants comparisons (see Table 4.4) were conducted to compare the Colavita effects observed Experiment 4.1 (in the Expect audition and Expect vision conditions), and those observed in Experiment 4.3 (in the Expect audition-Auditory exogenous cue, Expect audition-Visual exogenous cue, Expect vision-Auditory exogenous cue, and Expect vision-Visual exogenous cue conditions). The data were compared using between-participants ANOVAs with the within-participants factor of Response (Auditory-only or Visual-only), and the 122

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

between participants factor of Cue (Cue or No cue). As the only difference between the conditions in Experiments 4.1 and 4.3 was that exogenous cues were presented in Experiment 4.3 (whereas none were presented in Experiment 4.1), any differences in the magnitude of the Colavita effect between the experiments must reflect the ability of the exogenous cues to modulate the effect (given that participants were attending to a particular modality at the time).
Table 4.4. Table summarizing the results of the four between-participants ANOVAs conducted to compare the Colavita effects observed in Experiment 4.1 (in the Expect audition and Expect vision conditions), and those observed in Experiment 4.3 (in the Expect audition-Auditory exogenous cue, Expect auditionVisual exogenous cue, Expect vision-Auditory exogenous cue, and Expect vision-Visual exogenous cue conditions). The within-participants factor of the ANOVA was Response (Auditory-only or Visual-only), and the betweenparticipants factor was Cue (Cue or No cue).
The conditions for comparison No. Experiment 4.3 Experiment condition 4.1 condition 1 Expect visionExpect Visual vision exogenous cue ANOVA factors Response [F(1, 26) = 24.25, p < .001]. Participants made significantly more visualonly than auditory-only responses overall (13.0% vs. 3.1% of all bimodal trials). [F(1, 26) = 22.79, p < .001]. Participants made significantly more visualonly than auditory-only responses overall (10.5% vs. 3.0% of all bimodal trials). [F < 1, n.s.] Cue [F < 1, n.s.] Interaction between Response and Cue [F < 1, n.s.]

2

Expect visionAuditory exogenous cue

Expect vision

[F(1, 26) = 1.90, p = .180]

[F(1, 26) = 1.77, p = .195]

3

Expect auditionVisual exogenous cue

Expect audition

[F < 1, n.s.]

[F(1, 26) = 10.72, p = .003]. The trend for the Colavita visual dominance effect failed to reach significance in the Expect auditionVisual exogenous cue condition (11.1% visual-only vs. 3.6% auditory-only errors; t(11) = 1.78; p = .105). In contrast, auditory dominance was observed in the Expect audition condition (6.1% visual-only responses vs. 16.4% auditory-only responses; t(15) = 2.94, p = .010). [F < 1, n.s.]

4

Expect auditionAuditory exogenous cue

Expect audition

[F(1, 26) = 16.79, p < .001]. Participants made significantly more auditoryonly than visual-only responses overall (13.2% vs. 4.1% of all bimodal trials). That is, auditory dominance was observed.

[F(1, 26) = 1.73, p = .199]

Comparisons 1 and 2 (see Table 4.4) revealed that the magnitude of the Colavita effect observed in the Expect vision condition was not significantly different from those observed in the Expect vision-Visual exogenous cue condition or the Expect vision-Auditory exogenous cue condition. Likewise, Comparison 4 revealed that the magnitude of the reverse Colavita effect (auditory dominance) observed in the Expect audition condition was not significantly different from that observed in the 123

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

Expect audition-Auditory exogenous cue condition. Thus, in Comparisons 1, 2, and 4, the exogenous cues did not exert a significant effect on the magnitude of the Colavita effect (given that participants were directing their attention endogenously toward the auditory or visual modalities). The comparison between the Expect audition and the Expect audition-Visual exogenous cue conditions (Comparison 3, Table 4.4), however, revealed a significant interaction between Response and Cue, attributable to auditory dominance being observed in the Expect audition condition, whereas a non-significant trend toward visual dominance was observed in the Expect audition-Visual exogenous cue condition. Thus, the results of Comparison 3 show that the presentation of visual exogenous cues significantly modulated the extent to which auditory dominance manifested (given that participants were endogenously attending toward the auditory modality). In sum, the between-participants comparisons between the conditions in Experiments 4.1 and 4.3 revealed that auditory exogenous cues did not have a significant effect on the magnitude of the Colavita effect; they did not significantly attenuate the Colavita effect when participants were endogenously attending to the visual modality (Comparison 2, Table 4.4), nor did they significantly increase the auditory dominance observed when participants endogenously attended to audition (Comparison 4). Visual exogenous cues did not significantly increase the magnitude of the Colavita effect when participants were endogenously attending to the visual modality (Comparison 1), but they did reverse the auditory dominance observed when participants endogenously attended to audition (Comparison 3). Thus, the only condition in which visual exogenous cues modulated the magnitude of the Colavita

124

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

effect (as compared to when no cues were presented) was when participants were endogenously attending to the auditory modality.

4.4

GENERAL DISCUSSION
The results of the experiments reported in Chapter 4 provide show that the

Colavita effect is modulated by the modality toward which participants endogenously and/or exogenously direct their attention. The findings of Experiments 4.1 and 4.3 provide support for the long-standing notion that the Colavita effect may be caused by a tendency that participants ordinarily have to direct their attention endogenously toward the visual modality (cf. Colavita & Weisberg, 1979; Egeth & Sager, 1977; Posner et al., 1976). Furthermore, the results of the Experiments 4.2 and 4.3 demonstrate that visual stimuli are more effective than auditory stimuli at exogenously attracting a participant’s attention (also see Turatto et al., 2002), and suggest that participants attending exogenously toward vision may also be a factor that contributes to the emergence of the Colavita effect. The mechanisms by which attention (directed either endogenously or exogenously toward vision) may affect the Colavita effect will now be discussed.

4.4.1 The law of prior entry The law of prior entry (Titchener, 1908) can provide a mechanism for how directing attention, either endogenously or exogenously, toward a particular sensory modality can modulate the Colavita effect (also see Egeth & Sager, 1977, for a brief explanation of how prior entry could contribute to the Colavita effect). According to the law of prior entry, attended stimuli are perceived more rapidly than 125

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

simultaneously-presented stimuli that are not attended (e.g., James, 1890; Mollon & Perkins, 1996; Posner, Nissen, & Ogden, 1978; Titchener, 1908; also see Spence et al., 2001b, for a review), where participants’ attention can be manipulated either exogenously or endogenously (Shore, Spence, & Klein, 2001). Thus, participants attending to a particular sensory modality (either through manipulations of exogenous or endogenous attention) tend to perceive stimuli presented in that modality as having been presented earlier in time than if they were not attending to that modality (or if they were attending to another sensory modality). Several researchers have demonstrated that participants have a general bias to endogenously attend toward the visual modality (Hohnsbein et al., 1991; Klein, 1977; Posner et al., 1976), even under conditions of divided attention (Spence et al., 2001b; Zampini et al., 2005b). In addition, it has been demonstrated in the present chapter (also see Turatto et al., 2002) that visual stimuli are more effective at exogenously capturing attention than auditory stimuli are. Thus, participants could well have a tendency to have their attention directed endogenously and/or exogenously toward vision during the performance of the Colavita task. This tendency to attend toward vision could potentially result in the prior entry of the visual stimulus (i.e., a delayed perception of the auditory stimulus), which in turn may contribute to participants responding more rapidly and preferentially to the visual component of bimodal targets. One of the aims of the experiment reported in Chapter 5 was therefore to investigate whether prior entry of the visual stimulus (or a delayed perception of the auditory stimulus) can provide an explanation for how participants having their attention directed toward the visual modality contributes to the Colavita effect. The design of the experiment was similar to that of Experiment 2.1, except that the

126

CHAPTER 4 - ATTENTION AND THE COLAVITA EFFECT

auditory and visual components of the bimodal stimuli were presented asynchronously (i.e., the order in which the auditory and visual stimuli were presented, and their temporal separation, was varied). If the Colavita effect is modulated by varying which component of the bimodal stimulus is presented first (i.e., one would predict the effect to be larger when the visual stimulus is presented first), and is eliminated when the auditory stimulus leads, then the prior entry of the visual stimulus could provide a plausible explanation for how directing attention toward the visual modality can contribute to the Colavita effect.

127

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

CHAPTER 5
5.0 DOES AUDIOVISUAL ASYCHRONY MODULATE THE COLAVITA EFFECT?
The Colavita effect bears some interesting parallels to the clinical phenomenon of extinction observed in neuropsychological patients (e.g., Bender, 1952; Bueti et al., 2004; di Pellegrino et al., 1997b; Frassinetti et al., 2002; Mattingley et al., 1997; Rapp & Hendel, 2003; Sarri et al., 2006; as discussed in Chapter 1, Section 1.7). In the Colavita effect, the presentation of visual stimuli appears to extinguish participants’ awareness of auditory stimuli (on a proportion of the bimodal trials), whereas in the case of extinction, it is typically the ipsilesional stimulus that reliably extinguishes patients’ awareness of the contralesional stimulus (on dual stimulation trials). The resemblance between the Colavita effect in normal participants and the phenomenon of crossmodal extinction observed in

neuropsychological patients may make the insights gained in previous studies of extinction particularly relevant to uncovering the mechanisms involved in the Colavita effect. As researchers have recently demonstrated that the extinction observed in clinical patients can be modulated by the temporal separation between, and temporal order of, the presentation of the ipsilesional and contralesional stimuli,

128

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

the question of whether the Colavita effect can be similarly modulated by these factors will now be explored in this chapter. The results of the studies investigating extinction suggest that extinction is maximal when the stimuli are presented physically simultaneously1, and reduces in a temporally graded manner as a function of any increase in the temporal separation between them (see Baylis, Simon, Baylis, & Rorden, 2002; Bueti et al., 2004; Costantini et al., 2007; di Pellegrino, Basso, & Frassinetti, 1997a; Guerrini, Berlucchi, Bricolo, & Aglioti, 2003; Karnath, Himmelback, & Rorden, 2002; Rorden, Mattingley, Karnath, & Driver, 1997; though see Cate & Behrmann, 2002). This suggests that extinction is not an all-or-nothing phenomenon, but rather that the influence of one stimulus on the other diminishes continuously the further apart the stimuli are presented in time. Research on patients suffering from extinction and neglect has also shown that they tend to suffer from difficulties in temporal processing; in particular, patients suffering from extinction have an abnormal bias toward reporting the contralesional stimulus as lagging behind the ipsilesional stimulus when they are presented simultaneously (e.g., Baylis et al., 2002; Guerrini et al., 2003; Sinnett, Juncadella, Rafal, & Soto-Faraco, 2007) by around 200ms (Rorden et al., 1997). It has been argued that the delayed processing of the contralesional stimulus may contribute to extinction patients’ difficulty in reporting it (Bueti et al., 2004; di Pellegrino et al., 1997a).

Note, though, that while most researchers have reported extinction effects to be greatest when the ipsilesional and contralesional stimuli are presented simultaneously, other researchers have found extinction to be larger when the ipsilesional stimulus leads than when the contralesional stimulus leads (e.g, Cate & Behrmann, 2002). These differences could be attributed to the fact that only a small number of patients participated in these studies (e.g., one extinction patient was tested in Cate and Behrmann’s, 2002, study). Therefore, individual differences between the patients may have contributed to the different outcomes reported in the various studies (cf. Stone, Porrill, Wood, Keeler, Beanland, Port, & Porter, 2001; their study will be discussed later in this chapter).

1

129

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Hence, extinction is modulated by the temporal separation between the ipsilesional and contralesional stimuli, and could also be attributed to deficits in the temporal processing of the stimuli (i.e., due to participants perceiving the contralesional stimulus as lagging behind the ipsilesional stimulus). One aim of the experiment reported in this chapter was therefore to investigate whether the magnitude of the Colavita effect would also be influenced by the temporal order and separation with which the auditory and visual components of the bimodal stimulus were presented. If the presentation of the first stimulus interfered with participants’ perception of and/or response to a second stimulus, then one would expect the Colavita effect to be larger on those trials in which the visual stimulus was presented slightly before the auditory stimulus on bimodal trials, and a reversed Colavita effect (cf. Experiments 4.1 and 4.3) to occur when the auditory stimulus preceded the visual stimulus. Given that extinction effects are reduced as the temporal separation between the stimuli increases, one might predict that any Colavita visual dominance effect would also decline as the interval between the two stimuli was increased (for example, at temporal separations at which participants can begin to reliably perceive and respond to the auditory and visual stimuli as distinct perceptual events). In support of this prediction, previous research has shown that a number of audiovisual interactions depend on the degree of temporal separation between the auditory and visual components of the bimodal stimulus; for example, the McGurk effect (e.g., Munhall et al., 1996) and the audiovisual stream/bounce illusion (Sekuler, Sekuler, & Lau, 1997; Watanabe & Shimojo, 2001) are larger the smaller the degree of temporal separation between the auditory and visual stimuli. On the other hand, however, there is a wealth of evidence to suggest that the detection of a stimulus can be enhanced

130

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

when it is presented at the same time as a stimulus from another modality (see Calvert, Spence, & Stein, 2004; Fort, Delpuech, Pernier, & Giard, 2002 for a review; also see Calvert, Brammer, Bullmore, Campbell, Iversen, & David, 1999; Calvert, Campbell, & Brammer, 2000; Driver & Spence, 2004; Frassinetti, Bolognini, & Làdavas, 2002; Odgaard, Arieh, & Marks, 2004; Risberg & Lubker, 1978; Stein & Meredith, 1993). Given that the phenomenon of extinction and certain visual dominance interactions have been shown to decrease as the temporal interval between the stimuli increases (as discussed), it is plausible to predict that the Colavita effect would, too, decrease in magnitude as the temporal separation between the auditory and visual stimuli increases. The second aim of Experiment 5.1 was to determine whether participants have a generally delayed perception of the auditory stimulus relative to the visual stimulus (cf. the phenomenon of extinction, where extinction patients have a delayed perception of the contralesional stimulus relative to the ipsilesional stimulus). By analogy with the literature on clinical extinction, one possible explanation for the Colavita visual dominance effect might be in terms of normal participants having a generally delayed perception of the auditory stimulus on bimodal trials (even though the two stimuli have always been presented simultaneously in all previous studies of the effect). Participants’ perception of the temporal order of auditory and visual stimuli has often been measured using crossmodal temporal order judgement (TOJ) tasks (e.g., Zampini et al., 2003a)2. In crossmodal TOJ studies, participants typically have to make unspeeded judgments regarding the relative order in which pairs of near-

2

Note, though, that simultaneity judgment tasks are also used to determine the extent to which participants perceive asynchronously presented auditory and visual stimuli as being presented simultaneously or not (e.g., Stone et al., 2001; Zampini et al., 2005a).

131

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

simultaneous stimuli from different sensory modalities are presented. This methodology has enabled researchers to calculate the average amount of time by which one stimulus has to be presented before another in order for the two to be perceived as simultaneous (the point of subjective simultaneity; PSS). A participant’s perception of audiovisual synchrony (i.e., the point at which the PSS is achieved) depends on a number of factors, such as the relative intensity of the stimuli used, and the relative and absolute spatial positions from which they are presented (factors which have been shown to affect the PSS; see Jaśkowski, 1999; King, 2005; Spence et al., 2001b). In addition, the PSS is subject to a great deal of individual variation (Stone et al., 2001)3. Consequently, in order to determine whether the Colavita effect observed in the experiments reported in this thesis was due to participants having a generally delayed perception of the auditory stimulus on the bimodal trials (due to the particular stimulus parameters used), it was necessary to measure participants’ PSS values using the same stimulus parameters as used in the Colavita task itself. The participants’ PSS values were therefore measured using a crossmodal TOJ task in which participants were presented with the same auditory and visual stimuli as those used in the Colavita task. There were two main goals for Experiment 5.1: First, in order to investigate whether the Colavita effect is modulated by the temporal order and separation of the auditory and visual components of the bimodal stimulus, the auditory and visual components of the bimodal stimulus were presented at a range of stimulus onset asynchronies (SOAs = ±600ms, ±300ms, ±150ms, ±75ms, and ±35ms; where

Stone et al. (2001) measured the PSS of different participants and reported that although the PSS values were significantly different between participants, the PSS values of individual participants were stable across testing sessions. Stone et al. therefore argued that the perception of simultaneity may be reliably different for different people. In fact, it was just such individual differences that gave rise to the study of Experimental Psychology in the first place (Mollon & Perkins, 1996).

3

132

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

negative SOAs indicate that the auditory stimulus was presented first) in the Colavita task. The SOA indicates the temporal separation between the onsets of two stimuli and their temporal order4. If the presentation of the first component (of the bimodal stimulus) were to interfere with participants’ response to the second component, then one might predict the Colavita effect to be larger when the visual stimulus leads (than when the auditory stimulus leads) and possibly reversed when the auditory stimulus leads. One might also expect the magnitude of the Colavita effect to be influenced by the temporal separation between the two stimuli. The second aim of Experiment 5.1 was to examine whether the Colavita effect could be attributed to participants having a generally delayed perception of the auditory stimulus (relative to the visual stimulus) on the bimodal trials. The participants’ PSS values were therefore measured using a crossmodal TOJ task. In Experiment 5.1, participants first completed a TOJ task, after which they performed a version of the Colavita task. The Colavita task was similar to that used in Experiment 2.1, with the exception that the auditory and visual components of the bimodal stimuli were presented at a range of different SOAs (rather than all being presented simultaneously).

5.1

EXPERIMENT 5.1

5.1.1 Methods Participants. 22 naïve participants (mean age of 23 years, age range from 1835 years; 10 males and 12 females) took part in Experiment 5.1. All except one of the participants were right-handed by self-report. The experimental session lasted for approximately 35 minutes.
4

Hence, an SOA of -35ms indicates that the auditory stimulus was presented 35ms before the visual stimulus.

133

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Apparatus and materials. These were exactly the same as in Experiment 2.1 with the exception that on the bimodal trials, the auditory and visual stimuli were presented at one of 10 SOAs (±600ms, ±300ms, ±150ms, ±75ms, and ±35ms). All of the participants performed the TOJ task followed by the Colavita task. The method sections of the two parts of the experiment are reported separately.

5.1.1.1 TOJ task methods Procedure. The participants were informed that the auditory and visual stimuli would be presented asynchronously on each and every trial. They were instructed to press one response key whenever the auditory target appeared to have been presented first, and another key whenever the visual target appeared to have been presented first, with the allocation of the stimuli to the response keys (the ‘y’ and ‘h’ keys) counterbalanced across participants. On each trial, the participants were presented with an auditory and a visual stimulus at one of the 10 SOAs. The next trial began 1600ms after the participant had responded. The task was completely unspeeded and the participants were instructed to respond as accurately as possible. No feedback regarding the correctness of the participant’s responses was provided. Design. The participants completed one block of 150 trials (15 trials per SOA). The order of stimulus presentation was randomised within the block of trials. A block of 30 practice trials was presented before each task. The practice trials were identical to the main experimental trials but were not analysed.

134

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

5.1.1.2 Colavita task methods Procedure. The procedure was exactly the same as in Experiment 2.1 with the following exceptions. The participants were informed that the auditory and visual targets would be presented asynchronously on all of the bimodal trials but were encouraged not to delay their responses even though there might sometimes appear to be a delay between the presentation of the auditory and visual components of the bimodal targets. The target stimuli were presented at the start of the trial (where the target could be a single unimodal stimulus, or else the presentation of both an auditory and a visual stimulus separated by one of 10 SOAs). The responses were collected from the onset of the stimulus that was being responded to (i.e., on bimodal trials, the auditory responses were collected from the onset of the auditory stimulus, and the visual responses from the onset of the visual stimulus). The next trial began automatically 1600ms after the start of the previous trial. Design. The participants were presented with 4 blocks of 200 trials. There were 80 visual, 80 auditory, and 40 bimodal trials in each block (with an equal number of trials being presented at each SOA).

5.1.2 Results 5.1.2.1 TOJ task results The data obtained from the TOJ task were used to calculate two measures, the Just Noticeable Difference (JND) and the PSS (the point of subjective simultaneity). The JND provides a standardized measure of the sensitivity with which participants can judge the temporal order of the two stimuli presented, it is the smallest temporal interval between two stimuli needed for participants to discriminate their temporal order correctly on 75% of the trials. The PSS provides an estimate of the time interval 135

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

by which a stimulus in one sensory modality would have to lead a stimulus in another modality in order for synchrony to be perceived. That is, it is the point at which participants would have made the ‘audition first’ and ‘vision first’ responses equally often (see Figure 5.1A).
Table 5.1. Table showing the percentage of ‘vision first’ responses and the ztransformed proportion scores, as a function of SOA for a single participant. The slope and intercept of the best-fitting line of the z-transformed data as a function of the SOA can be used to calculate a participant’s JND and PSS values. SOAS Percentage of ‘vision first’ responses (%) The z-transformed proportion scores -150 1 -75 6 -35 15 35 53 75 67 150 94

-2.33

-1.55

-1.04

0.08

0.44

1.55

For each participant, the percentage of ‘vision first’ scores was transformed to represent the proportion of ‘vision first’ trials. The proportion scores of ‘vision first’ responses were then converted to their equivalent z-scores using probit analysis under the assumption of a cumulative normal distribution (Finney, 1964). The transformed z-scores were obtained by applying the inverse of the standard normal distribution function to the proportion scores. This transformation allows one to perform a linear regression on the z-transformed data (see Figure 5.1B); using the slope and intercept of the best-fitting line of the z-transformed data, one can derive values for the PSS and JND. The JND is calculated by dividing the slope of the best-fitting line by 0.675, the reason being that the ±0.675 point corresponds to the 75% and 25% points on the cumulative normal distribution. The PSS represents the point at which the participants would have been predicted to make ‘vision first’ and ‘audition first’ responses equally often (i.e., the point at which the proportion of ‘vision first’ responses was 0.5) and is calculated as [PSS = -intercept/slope]. See Coren, Ward, and Enns (2004) for an introduction to the psychophysical methods used here. 136

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

SOA used to calculate the PSS

A
100

SOAs used to calculate the JND

% ‘Vision first’ responses

90 80 70 60 50 40 30 20 10 0 -200

The PSS provides an estimate of the point at which participants make ‘vision first’ and ‘audition first’ responses equally often

-150

-100

-50

0

50

100

150

200

Audition first

SOA (ms)

Vision first

Proportion of ‘Vision first’ responses

B
0.99 0.92 0.75 0.50 0.25

The JND is calculated by dividing the slope of the best fitting line by 0.675; since ±0.675 represents the 75% and 25% point on the cumulative normal distribution

0.08

0.01 -200

-150

-100

-50

0

50

100

150

200

Audition first

SOA (ms)

Vision first

Figure 5.1. The Point of Subjective Simultaneity (PSS; A) and the Just Noticeable Difference (JND; B) measures derived from the TOJ task, and the z-transformed data (B).

For each participant, the z-transformed data from the ±150ms, ±75ms, and ±35ms SOAs were used to calculate the best-fitting straight line, which was used to derive values for the slope and intercept, which, in turn, were used to calculate the JND and PSS values. The ±300ms and ±600ms points (which were primarily relevant for the Colavita task) were excluded from this analysis because most participants performed near-perfectly at these intervals, and so no additional variance was accounted for by these data points (see Navarra, Vatakis, Zampini, Soto-Faraco, Humphreys, & Spence, 2005; Spence et al., 2001b, on this point). Participants were excluded from the data analysis if their JNDs exceeded ±150ms; see Zampini et al.,

137

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

2005b, for similar exclusion criteria). This resulted in the exclusion of 8 participants from the data analyses. The analysis of the TOJ data revealed that the visual stimulus had to lead the auditory stimulus by 12ms (standard error = 9ms) in order for the PSS to be achieved. Thus, if the auditory and visual stimuli had been presented simultaneously the participants would have tended to perceive the auditory stimulus as having been presented, if anything, slightly earlier in time than the visual stimulus. Importantly, however, for the particular stimuli used in Experiment 5.1 (and for the particular participants tested; cf. Stone et al., 2001), this value was not significantly different from 0ms (t(13) = 1.33, p = .206). The mean JND was 77ms (standard error = 5ms). These values are consistent with the results reported in several other recent studies (e.g., Spence et al., 2001b; Spence, Baddeley, Zampini, James, & Shore, 2003; Zampini, Shore, & Spence, 2003a, b).

5.1.2.2 Results of the Colavita task Participants failed to make any response on 0.2% of the trials in the Colavita task and these trials were not included in the data analyses. The RT and error data from Experiment 5.1 are shown in Table 5.2. Error data. The data from the bimodal trials in which the participants responded incorrectly (see Figure 5.2A) were analysed using an ANOVA with the factor of Response (Auditory-only or Visual-only), First Modality (Audition first or Vision first), and SOA (±600ms, ±300ms, ±150ms, ±75ms, or ±35ms). The analysis revealed a significant main effect of Response [F(1, 13) = 5.44, p = .036], attributable to the participants making significantly more visual-only than auditory-only responses (8.2% vs. 4.4% of all bimodal trials, respectively), thus demonstrating a significant 138

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Colavita effect overall. The analysis also revealed a significant interaction between Response and First Modality [F(1, 13) = 5.11, p = .042]. In particular, the participants made significantly more visual-only responses than auditory-only responses (i.e., there was a significant Colavita effect) when the visual stimulus was presented first (11.6% visual-only vs. 1.4% auditory-only responses; t(13) = 2.52, p = .026), but not when the auditory stimulus was presented first (4.8% visual-only vs. 7.4% auditoryonly responses; t(13) = 1.16, p = .267). A significant Colavita effect was observed at the -35ms, +35ms, +75ms, and +150ms SOAs, while at the -600ms SOA, a reverse Colavita effect was observed (see Table 5.2, Figure 5.2B, and Figure 5.2C). The numerically largest Colavita effects were observed at the +75ms and +150ms SOAs.
Table 5.2. Mean error rates for the bimodal target stimuli in the Colavita task. The auditory and visual stimuli were presented at one of 10 stimulus onset asynchronies (SOAs; ±600ms, ±300ms, ±150ms, ±75ms, and ±35ms; where negative SOAs indicate that the auditory stimulus was presented first). Mean reaction times (RTs; ms) for correct responses to bimodal target stimuli. Standard errors are shown in parentheses.
First modality SOA (ms) Error rates (%) Auditory-only responses Visual-only responses RTs (ms) Auditory responses Visual responses Audition -600 -300 -150 -75 -35 +35 +75 +150 +300 Vision +600

14.6 (6.1) 1.0 (1.0)

10.8 (5.4) 1.9 (1.1)

4.5 (2.2) 6.3 (2.4)

3.1 (1.3) 6.7 (2.7)

4.0 (1.6) 8.5 (2.8)

1.8 (1) 9.9 (2.5)

0.9 (0.9) 13.2 (4.4)

0.0 (0.0) 11.6 (4.5)

2.2 (1.2) 13.4 (6.1)

1.8 (1) 9.8 (6.5)

470 (46) 503 (18)

506 (46) 508 (24)

515 (35) 516 (25)

552 (30) 503 (24)

564 (32) 501 (27)

587 (33) 510 (29)

584 (31) 495 (25)

529 (24) 506 (30)

497 (22) 481 (37)

497 (39) 433 (21)

139

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

A
25 20

Errors (%)

15

10

5

0 -800

-600

-400

-200

0

200

400

600

800

Audition first

SOA (ms)

Vision first

B
25

Errors (% bimodal trials)

20 15 10 5 0 -5 -800

*

Visual-only responses

Auditory-only responses

*

** ** * **
p < .05 p < .01

-600

-400

-200

0

200

400

600

800

Magnitude of the Colavita effect (%)

C

Audition first

SOA (ms)

Vision first

20 15 10 5 0 -5 -10 -15 -20 -800

-600

-400

-200

0

200

400

600

800

D

Audition first

SOA (ms)

Vision first

650 600

Auditory responses

Visual responses

RT (ms)

550 500 450 400 -20 -800 -800
Audition first

-600 -600

-400 -400

-200 -200

0 0

200 200

400 400

600 600

800 800

SOA (ms)

Vision first

Figure 5.2. Summary of the mean percentages of errors on the bimodal trials in the Colavita task (A), the mean percentages of auditory-only and visual-only errors in the Colavita task (B), and the mean magnitude of the Colavita effect in the Colavita task (the percentage of visual-only errors minus the percentage of auditory-only errors; C), and the mean RTs to the auditory and visual components of the bimodal stimulus (for correct responses; D), plotted as a function of the stimulus onset asynchrony (SOA) between the auditory and visual stimuli in Experiment 5.1. The error bars indicate the standard errors of the means.

140

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Note that although there appear to be numerically large Colavita effects at the +300ms and +600ms SOAs than at the smaller SOAs of ±35ms, the reason why the Colavita effect failed to reach significance at these SOAs is because of the large standard errors (see Figure 5.2B and Table 5.2). For example, the high percentage of visual-only errors observed at the +600ms SOAs was driven by the exceptionally high visual-only error rates of just two of the participants. In fact, 8 of the 14 participants made no visual-only or auditory-only errors at all at the +600ms SOA, suggesting, firstly, that participants were able to respond accurately to the two components of the bimodal stimulus when they were far enough apart, and second, that the seemingly large Colavita effect reported at the +600ms SOA was caused by those two particular participants having a tendency not to respond to the second stimulus on those trials. Thus, it can be concluded that the Colavita effect was eliminated at the +300ms and +600ms SOAs. Finally, there was a significant interaction between Response and SOA [F(4, 52) = 5.62, p = .001], attributable to the fact that the Colavita effect was significant overall at temporal separations (between the two stimuli) of ±35ms5 and ±75ms (magnitude of the Colavita effect = 6.2% and 7.9%; t(13) = 3.98, p = .002; t(13) = 2.57, p = .023), and borderline-significant at temporal separations of ±150ms (mean Colavita effect = 6.7%; t(13) = 2.14, p = .052). The fact that the average Colavita effect was significant at the smaller temporal separations (±35ms and ±75ms), borderline significant at the temporal separations of ±150ms, and insignificant at the larger ±300ms and ±600ms temporal separations suggests that the temporal separation between the auditory and visual components of the bimodal stimulus influences

Note that the value given for the Colavita effect present at the 35ms temporal separation is the average of the Colavita effect at the -35ms SOA and the Colavita effect at the 35ms SOA. The same applies to the values given for the Colavita effect at the 75ms and 150ms temporal separations.

5

141

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

whether or not the Colavita effect will occur. None of the other terms in this analysis reached significance: for First Modality [F < 1, n.s.], SOA [F < 1, n.s.], First Modality × SOA [F(4, 52) = 1.37, p = .258], and for First Modality × SOA × Response [F(4, 52) = 1.91, p = .121]. Participants made 5.4% errors on the unimodal auditory and 8.0% errors on the unimodal visual trials. An ANOVA conducted on the error data with the factor of Stimulus Type (Auditory, Visual, or Bimodal) did not reveal a significant main effect [F(1.11, 14.37) = 1.85, p = .196]. RT data. On the unimodal target trials, participants took an average of 440ms to respond to the unimodal auditory stimuli and 410ms to respond to the unimodal visual stimuli. The RT data from those trials in which the participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual) and Target Type (Unimodal or Bimodal). This analysis revealed that participants responded significantly more rapidly on the unimodal (425ms) than on the bimodal target trials (513ms), resulting in a significant main effect of Target Type [F(1, 11) = 136.45, p < .001]. There was also a significant main effect of Target Modality [F(1, 11) = 8.36, p = .013], with participants responding more rapidly to visual (452ms) than to auditory targets (485ms), overall. There was, however, no interaction between Target Modality and Target Type [F < 1, n.s.]. Next, the RT data from the bimodal trials were analysed. (Remember that for the bimodal target trials, auditory response latencies were measured from the onset of the auditory stimulus, whereas the visual response latencies were measured from the onset of the visual stimulus.) The data were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), First Modality (Audition first or Vision first), and SOA (±600ms, ±300ms, ±150ms, ±75ms, or ±35ms). This analysis

142

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

revealed a significant main effect of Target Modality [F(1, 13) = 9.62, p = .008], with the participants responding significantly more rapidly to visual (496ms) than to auditory targets (530ms). There was also a significant main effect of SOA [F(1.95, 25.37) = 6.66, p = .005], attributable to the participants responding more rapidly on the bimodal trials as the temporal separations between the stimuli increased (mean RTs = 541ms, 533ms, 516ms, 498ms, and 476ms, for stimuli presented at the temporal separations of ±35ms, ±75ms, ±150ms, ±300ms, and ±600ms, respectively). The participants also responded more rapidly to stimuli presented at temporal separations of ±600ms than at temporal separations of ±35ms, ±75ms, ±150ms, and ±300ms (t(13) = 3.66, p = .003; t(13) = 3.13, p = .008; t(13) = 2.86, p = .013; t(13) = 2.16, p = .050), and less rapidly to stimuli presented at temporal separations of ±35ms than at temporal separations of ±150ms or ±300ms (t(13) = 2.46, p = .029; t(13) = 2.54, p = .025). Finally, there was a significant interaction between Target Modality and SOA [F(2.41, 31.38) = 3.44, p = .037] (see Figure 5.2D). Subsequent t-tests revealed that this term reflected the fact that the participants responded significantly more rapidly to the visual than to the auditory component of the bimodal targets at temporal separations of ±35ms and ±75ms (mean differences = 70ms and 69ms, respectively; t(13) = 5.78, p < .001; t(13) = 4.44, p = .001), whereas this difference failed to reach significance at any of the other separations (mean differences = 11ms, 8ms, and 15ms, at temporal separations of ±150ms, ±300ms, and ±600ms, respectively; t(13) = .65, p = .530; t(13) = .42, p = .681; t(13) = .54, p = .600). The interaction between First Modality and SOA failed to reach statistical significance [F(4, 52) = 1.71, p = .161]. None of the other terms in the analysis of the RT data reached significance: for First Modality [F < 1, n.s.], for the interactions between First

143

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Modality × Target Modality [F(1, 13) = 1.70, p = .215], and for First Modality × Target Modality × SOA [F(4, 52) = 1.03, p = .300].

5.1.3 Discussion A significant Colavita effect was demonstrated overall; that is, when participants made an error on the bimodal target trials (which they did on 12.6% of all bimodal trials), they made significantly more visual-only than auditory-only responses (8.2% vs. 4.4% of all bimodal trials, respectively).

5.1.3.1 The Colavita effect and prior entry The Colavita effect was significant when audition led by 35ms and when vision lead by 35ms, 75ms, and 150ms. In addition, there was also a non-significant trend toward more visual-only than auditory-only responses at auditory leads of 75ms and 150ms, and at visual leads of 300ms and 600ms. The results of Experiment 5.1 therefore suggest that the Colavita effect is affected by the temporal order in which the auditory and visual stimuli are presented and/or perceived. The fact that the Colavita effect was significantly larger when the visual stimulus was presented first, and reversed or attenuated (depending on the SOA) when the auditory stimulus was presented first, suggests that prior entry (Titchener, 1908) could provide a plausible explanation for the Colavita effect (as discussed in Chapter 4, General discussion). This point will be discussed in the General discussion of this chapter.

144

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

5.1.3.2 The RT data The analysis of the RT data from the Colavita task revealed that the participants responded significantly more rapidly to the visual than to the auditory stimuli overall. In contrast to the RT data, however, the results of the TOJ task showed that for the particular stimuli and parameters used here, as well as in all of the other studies in this thesis (except for the experiments reported in Chapters 7 and 8 in which different stimuli from those presented in Experiment 5.1 were presented) the PSS (a visual lead of 12ms) was not significantly different from zero. Furthermore, if anything, the participants tended to perceive the auditory stimulus, and not the visual stimulus, as having been presented first. This result therefore suggests that the Colavita effect observed in the studies reported in this thesis (in which the auditory and visual stimuli were actually presented simultaneously) was not due simply to participants having a generally delayed perception of the auditory stimulus relative to the visual stimulus. The next result in need of an explanation is the dissociation between the RT and TOJ data in Experiment 5.1; that is, participants tended to perceive the auditory stimulus before the visual stimulus, but responded faster to the visual than to the auditory stimulus on bimodal trials. It is the case that similar dissociations between the RT and TOJ data have frequently been documented over the years (see Miller & Schwarz, 2006, for a recent review), and it has been suggested that such dissociations may be due to differences in the criterion for responding adopted by participants to accommodate the different task demands that are associated with the performance of RT and TOJ tasks (see Miller & Schwarz, 2006). The analysis of the RT data also revealed that participants responded more rapidly overall to bimodal stimuli as the temporal separation between the auditory and

145

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

visual components increased (remember that the auditory and visual RTs were measured from the onset of the auditory and visual stimuli, respectively; see Figure 5.2D). There are at least two possible factors that may have contributed to this effect. The first explanation for the fact that participants responded more rapidly overall to bimodal stimuli presented at longer SOAs than at shorter SOAs is in terms of an alerting effect (see Posner, 1978; Spence & Driver, 1997a). That is, the presentation of the first stimulus may have made the participants more alert overall, thus resulting in them responding more rapidly to the second stimulus. The alerting effect may have built up over time (at least over the range of SOAs tested in Experiment 5.1), thus resulting in a larger alerting effect for stimuli presented at the larger SOAs. The second explanation is that at the longest SOAs participants may have been responding to the first stimulus as if it were a unimodal target and responded to the second stimulus when they later noticed that it had been presented. Support for this idea comes from the results of t-tests which revealed that there were no significant differences in response latencies to unimodal stimuli and to the first stimulus presented on bimodal trials at the 600ms SOA, for auditory targets (unimodal auditory RT vs. auditory RT at the -600ms SOA where the auditory stimulus was presented first; t(13) = 1.51, p = .271) or visual targets (unimodal visual RT vs. visual RT at the +600ms SOA; t(13) = 1.29, p = .217). At the smaller SOAs, however, the differences between the unimodal and bimodal RTs became numerically larger (see Figure 5.2D) and these differences were significant at the -150ms, -75ms, -35ms, +35ms, +75ms, +150ms, and +300ms SOAs (t(13) = 5.11, p < .001; t(13) = 5.16, p < .001; t(13) = 5.94, p < .001; t(13) = 6.15, p < .001; t(13) = 5.38, p < .001; t(13) = 5.22, p < .001; t(13) = 2.02, p = .051). Further support for the notion that participants may have been

146

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

responding to the first stimulus as if it were a unimodal target comes from inspection of Figures 5.3 and 5.4. Figure 5.3 (next page) contains histograms presenting the frequencies with which participants responded within a certain RT interval (e.g., between 300ms350ms, or between 350ms-400ms) for unimodal auditory and visual stimuli, and for the first stimulus presented on bimodal trials for the ±600ms, ±300ms, ±150ms, ±75ms, and ±35ms SOAs. In Figure 5.3, it appears that at larger temporal separations (e.g., at SOAs of ±600ms or ±300ms, in Figures 5.3A, B, F, and G) participants tended to respond to the first stimulus as if it were a unimodal stimulus; this is supported, first, by the observation that the distribution of RTs on the ±600ms SOA bimodal trials was similar to the distribution of RTs on unimodal trials, and second, by the fact that there was no significant difference in response latency between participants’ responses to the first stimulus on ±600ms SOA bimodal trials and their responses to unimodal stimuli. In other words, on trials in which there were large temporal separations (e.g., separations of 300ms and 600ms) it is possible that participants may have responded to the auditory and visual components of the bimodal stimuli as they would have responded to two sequentially presented unimodal stimuli. At smaller temporal separations (e.g., at SOAs of ±150ms, ±75ms, or ±35ms, in Figures 5.3C, D, E, H, I, and J) the distribution of responses to the first component of the bimodal stimulus became more dissimilar to the distribution of responses to unimodal stimuli (there is a tendency towards the distribution having two peaks at smaller temporal separations); this suggests that participants may have been delaying their responses to the first component of the bimodal stimulus.

147

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Auditory RTs

Visual RTs
1000 900

A
Frequency
Frequency

Unimodal

900 900

B
800 700 600

Unimodal

800800
700 700

800900

600600 500500
400 400 300 300

Frequency Frequency Frequency
1000 1100 400 1200

700

600

500
500

400
400

300 300

200

200

200 200 100 100
1000 1600 1400 1100 200 500 600 800 400 1300 1200 300 700 900 200 500 800 1400 1500 1300 1500 1500 1500 1400 1500 1600 1600 1600 1500 1600 1600 1600 300 600 700 900

100 100

00

00

Distribution of RTs (ms)

Distribution of RTs (ms)

C
50 50 45 45 40 40

Histogram

-600ms SOA

50 50 45 45 40 40

H

Histogram

600ms SOA

Frequency Frequency

35 35
30 30 25 25

Frequency Frequency

35 35
30 30

25 25

20 20 15 15
10 10

20 20 15 15
10 10

55
1000 1100 400 1200 1400 500 900 1600 200 300 600 800 700 1300 1500

55
1000 1100 400 1200 200 500 800 1300 1300 1200 1300 1300 1400 1400 1300 1400 1400 300 600 700 900

00

00

D
50
50

Distribution of RTs (ms)
Histogram

-300ms SOA

I
50 50 45 45 40 40

Distribution of RTs (ms)
Histogram

-300ms SOA

45

45

Frequency Frequency Frequency

35 35 30 30 25

25

20 20 15 15

10 10

Frequency Frequency Frequency

40

40

35 35 30 30

25 25 20 20 15 15

10 10

55
1000 1100 400 1200 200 300 500 800 1400 1300 1500 1600 600 700 900

55
1000 400 1100 1100 1000 1100 1200 1200 1100 1200 200 500 800 300 600 700 900 800 900 900

00

00

Distribution of RTs (ms)

Distribution of RTs (ms)

E
50 50 45 45 40 40 35 35 30 30 25 25

-150ms SOA

Histogram

50 50 45 45 40 40

J

-150ms SOA

Histogram

Frequency Frequency

Frequency

35 35
30 30 25 25

20 20
15 15 10 10

20 20 15 15
10 10

55
1000 1100 200 400 800 1200 1400 300 500 1300 1600 600 900 700 1500

55
1000 1000 900 200 400 300 500 600 700 800 700 800

00

00

Distribution of RTs (ms)

Distribution of RTs (ms)

F
50 50 45 45 40 40

-75ms SOA

Histogram

IK
50 45
50

-75ms SOA

H i s t ogr a m

405 4

Frequency Frequency

35 35
30 30

Frequency

350 4
3 305

25

30 25 20

25

25

20

20 20 15 15
10 10

15
15

10 10 55
200 300 400 500 600

55
1000 1100 200 400 800 1200 1300 1400 300 500 1500 1600 600 700 900

00

00

Distribution of RTs (ms)

Distribution of RTs (ms)

G
50 50 45 45 40 40

-35ms SOA

Histogram

50 50 45 45 40 40 35 35 30 30 25 25

L

-35ms SOA

Histogram

Frequency Frequency

35 35
30 30 25 25

20 20 15 15
10 10

Frequency Frequency
1000 1100 200 400 500 600 800 900

20 20 15 15
10 10

55
1200 1400 1600 300 700 1300 1500

55
200 300 400 500 600 700 1500

00

00

Distribution of RTs (ms)

Distribution of RTs (ms)

Figure 5.3. Histograms showing the frequencies at which participants responded within a certain RT interval for unimodal auditory stimuli (A) and unimodal visual stimuli (B; top two histograms), for bimodal auditory stimuli presented at SOAs of 600ms (C), -300ms (D), -150ms (E), -75ms (F), and -35ms (G), and for bimodal visual stimuli presented at SOAs of +600ms (H), +300ms (I), +150ms (J), +75ms (K), and +35ms (L).

148

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Figure 5.4 summarises the mean RTs to the auditory and visual components of the bimodal stimulus, where the RT latencies displayed for both the auditory and visual responses represent the time that has passed since the onset of the first stimulus in the trials. Figure 5.4 shows that the larger the temporal separation between the auditory and visual components of the bimodal stimulus, the greater the temporal separation between participants’ auditory and visual responses.
1400

Auditoryresponses Auditory responses
Mean unimodal auditory RT

Visual responses Visual responses
Mean unimodal visual RT

1200

1000

RT (ms)

800

600

400

200

0 -800 -600 Audition first

-400

-200

0

200

400

600

SOA (ms)

800 Vision first

Figure 5.4. The mean RTs to the unimodal auditory and visual stimuli, and to the auditory and visual components of the bimodal stimulus, for correct responses, plotted as a function of the stimulus onset asynchrony (SOA). The RTs were measured from the onset of the first stimulus that was presented during the trial.

In sum, the pattern of RT data is consistent with the notion that at larger temporal separations (e.g., at SOAs of ±600ms or ±300ms) participants were more likely to respond to the two components of the bimodal stimulus sequentially and separately, whereas at smaller temporal intervals, participants appear to have had some difficulty in perceptually processing, or selecting the appropriate response toward, both components of the bimodal target at once (cf. Marois & Ivanoff, 2005).

149

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

5.1.3.3 The temporal separation between the auditory and visual stimuli It has been argued that our perceptual system can integrate asynchronouslypresented correlated multisensory inputs as long as they are presented within the boundaries of the temporal window for multisensory integration (e.g., Dixon & Spitz, 1980; Munhall et al., 1996; Spence & Squire, 2003; van Wassenhove, Grant, & Poeppel, 2002; see Navarra et al., 2005, for a recent review). Within this temporal window, it has been argued that people typically bind auditory and visual stimuli into a single multisensory event with a unique temporal onset. As many audiovisual interactions depend on the temporal binding of the constituent auditory and visual stimuli (as previously discussed in Section 5.0), it is possible that the binding of the auditory and visual components of the bimodal stimulus may also contribute to the Colavita effect. One possibility, based on the fact that some audiovisual interactions are larger the smaller the temporal separation between the auditory and visual stimuli (e.g., Fendrich & Corballis, 2001; Morein-Zamir et al., 2003; Munhall et al., 1996; Sekuler, Sekuler, & Lau, 1997; Watanabe & Shimojo, 2001), is that the Colavita effect should be larger when the auditory and visual stimuli are temporally bound. Another possibility, based on the findings that the detection of a stimulus can be enhanced when it is presented concurrently with a stimulus from another modality (e.g., Calvert et al., 1999; Calvert et al, 2000a, b; Fort et al., 2002; Odgaard et al., 2004; Risberg & Lubker, 1978), is that the Colavita effect should be smaller the closer together in time the auditory and visual stimuli are. An analysis was performed in order to determine whether the Colavita effect only occurs within the conventionallydefined temporal window of audiovisual integration. The temporal window of audiovisual integration was calculated for each participant, in terms of the PSS ± JND (see Vatakis & Spence, 2006, 2007b). In

150

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

Experiment 5.1, the visual stimulus would have had to lead by 12ms at the PSS. Given that the JND was 77ms, the temporal window of audiovisual integration for the particular auditory and visual stimuli used in the present study was calculated as lying between an auditory lead of approximately 65ms and a visual lead of 89ms (as shown schematically in Figure 5.5).
-65ms 89ms

Temporal window for audiovisual integration
100

12ms

% ‘Vision first’ responses

80

60

40

20

0 -200 Audition first

-100

0

100

SOA (ms)
PSS PSS ± JND

200 Vision first

Figure 5.5. The untransformed data representing the mean percentage of ‘vision first’ responses in the TOJ task, plotted as a function of the SOA between the auditory and visual stimuli in Experiment 5.1. The bar represents the putative temporal window for audiovisual integration. The PSS (12ms) represents the SOA at which participants are unsure of which stimulus was presented first, and the JND (77ms) represents the width of the temporal window of audiovisual integration (Navarra et al., 2005; see Section 5.1.4). The boundary of the temporal window of audiovisual integration is calculated as the PSS ± JND (Vatakis & Spence, 2006, 2007b). The error bars indicate the standard errors of the means.

The results of the Colavita task revealed that the Colavita effect was significant at SOAs (-35ms, +35ms, and +75ms) within the temporal window of integration (-65ms to +89ms), and at the SOA of +150ms (when vision led by 150ms; where participants should have reliably perceived the visual stimulus as having come

151

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

first), an SOA that fell far outside of the temporal window of audiovisual integration. The finding that the Colavita effect was significant at SOAs that were both within and outside of the temporal window of separation suggests that the temporal binding of the auditory and visual stimuli might not play such a critical role in the emergence of the Colavita effect. In addition, there was no graded increase or decrease in the magnitude of the Colavita effect as a function of increasing SOA. The Colavita effect was not significantly smaller or larger at the smaller SOAs than at larger SOAs; there was no significant difference between the magnitude of the Colavita effect at the -35ms SOA than at the +35ms, +75ms, and +150ms SOAs (t(13) = 1.43, p = .179; t(13) = 1.85, p = .083; t(13) = 1.63, p = .116), or any significant difference between the magnitude of the Colavita effect at the +35ms SOA than at the +75ms or +150ms SOAs (t(13) = 1.11, p = .283; t(13) = 1.09, p = .294). The finding that there was no graded change in the magnitude of the Colavita effect as the SOA increased, again, supports the notion that the extent to which participants temporally bind the auditory and visual stimuli may not contribute to the Colavita effect. It appears that it is, instead, the temporal order in which the auditory and visual stimuli are presented (where the Colavita effect is larger when vision leads) and the extent to which participants can respond sequentially to the auditory and visual stimuli (where the Colavita effect appears to be eliminated when the participant can respond to the auditory and visual stimuli as two sequentially presented unimodal stimuli) that determines whether the Colavita effect will occur. The analysis of the RT data suggests that at temporal separations of ±300ms and ±600ms participants were responding to the auditory and visual stimuli as if they were sequentially presented unimodal stimuli (in which case one would expect

152

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

participants to be able to respond accurately to both stimuli). One question left to answer is why, given this, there was a significant reverse Colavita effect at the -600ms SOA (when audition led by 600ms) and a numerically large trend toward the Colavita effect occurring at the +300ms and +600ms SOAs. One explanation for this pattern of results is that on some of those trials participants may have mistakenly assumed that they had responded to a unimodal stimulus, relaxed (or diverted their attention elsewhere until the next trial), and may therefore have been unprepared to respond to the second stimulus. This idea is supported by the finding that participants’ RTs to the first stimulus of a ±600ms bimodal stimulus were not significantly different from their RTs to unimodal stimuli, suggesting that participants did indeed respond to the first stimulus as if it were a unimodal stimulus. This effect of participants being unprepared to respond to the second stimulus may have been more detrimental on trials in which the auditory stimulus came first, because it is possible that after participants had responded to the auditory stimulus participants may not have seen the visual stimuli if they had momentarily looked away or blinked when the visual stimulus was presented. In contrast, on those trials in which the visual stimulus came first, participants would have been able to hear the second auditory stimulus whether or not they were fixating toward the target sources.

5.2

GENERAL DISCUSSION
The primary aims of Experiment 5.1 were first to investigate whether the

Colavita visual dominance effect could be attributed to participants perceiving the auditory stimulus as lagging behind the visual stimulus on bimodal trials, and second to determine whether the effect could be modulated by varying the temporal

153

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

separation and temporal order of the auditory and visual stimuli on the bimodal target trials. Three main findings emerged from the analyses reported here: First, participants’ performance on the TOJ task revealed that the PSS did not differ significantly from zero. Thus, for the particular stimuli used in Experiment 5.1, if the auditory and visual stimuli had been presented simultaneously, participants did not reliably perceive the visual stimulus as having come first when judging their temporal order of occurrence. This suggests that the Colavita effect observed in Experiment 5.1, and in the other studies reported in this thesis (which used the same stimulus parameters as used here; where the auditory and visual stimuli were always presented simultaneously on the bimodal trials) was probably not due simply to participants having a general6 tendency to perceive the auditory stimulus as lagging behind the visual stimulus. The second finding to emerge from the analysis of Experiment 5.1 was that the Colavita effect was modulated by the temporal separation between the auditory and visual stimuli; in particular, the Colavita effect was eliminated when the temporal separation was large enough that participants could respond sequentially to the stimuli (as if they were unimodal stimuli). However, the magnitude of the Colavita effect did not appear to be modulated by the extent to which participants temporally bound the auditory and visual stimuli; the Colavita effect was significant at SOAs both within and outside of the temporal window of integration. The third finding to emerge from the analysis of Experiment 5.1 was that the Colavita effect was modulated by the temporal order in which the auditory and visual stimuli were presented on the bimodal target trials. Specifically, the Colavita effect

The fact that participants’ PSS did not differ from zero on the TOJ task, however, does not rule out the possibility that participants would have a delayed perception of the auditory stimulus on at least some of the trials on the Colavita task. If participants had been attending to vision during the Colavita task, they would perceive the auditory stimulus as lagging behind the visual stimulus.

6

154

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

was larger when the visual stimulus was presented first, and reversed or attenuated (depending on the particular SOA) when the auditory stimulus was presented first. This finding can be explained in terms of the phenomenon of prior entry (Titchener, 1908).

5.2.1 Prior entry The law of prior entry states that attended stimuli are perceived more rapidly (and as having been presented earlier in time) than simultaneously-presented stimuli that are not attended (Titchener, 1908; also see Spence et al., 2001b, for a review; also see Chapter 4, General discussion, Section 4.4.1). The experiment reported in Chapter 4 showed that attention directed endogenously toward the visual modality appears to be necessary for the Colavita effect to occur, and attention directed exogenously toward the visual modality also contributes to the Colavita effect. Prior entry was put forward as an explanation for how directing attention toward a particular sensory modality could modulate the Colavita effect. In particular, it was proposed that if participants attended to the visual modality (either by endogenously directing their attention to vision, or by having their attention exogenously captured by the visual stimulus), they would have perceived the visual component (of the bimodal target) before the auditory component, which could have caused them to respond only to the visual component (and vice versa if participants were attending to the auditory modality). One of the aims of Experiment 5.1 was therefore to investigate whether the Colavita effect could be modulated by varying which component of the bimodal stimulus was presented first, and whether the effect could be eliminated when the auditory stimulus led. This would have helped to determine whether perceiving a 155

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

particular stimulus first (on bimodal trials) would increase the errors participants made in responding to the second stimulus. The results of Experiment 5.1 revealed that the Colavita effect was indeed larger when the visual stimulus was presented first, and reversed or attenuated (depending on the SOA) when the auditory stimulus was presented first. (It should be noted that the Colavita effect was no longer significant when the auditory and visual stimuli were so far apart so that participants could respond to them sequentially and separately.) The results of Experiment 5.1 therefore support the notion that the prior entry of the visual stimulus (i.e., a delayed perception of the auditory stimulus) can provide one explanation for the Colavita effect.

5.2.2 The unity effect The Colavita effect was modulated by the temporal separation of the stimuli. However, the finding that the Colavita effect was significant at SOAs both within and outside of the temporal window of audiovisual integration (although predominantly within the window), and the finding that there was no graded change in the magnitude of the Colavita effect as the SOA increased, suggests that this effect of temporal separation on the Colavita effect was not caused by the Colavita effect being modulated by the extent to which participants temporally bound the auditory and visual stimuli. Rather, the results of Experiment 5.1 suggest that this effect of temporal separation on the Colavita effect was due to the fact that when the auditory and visual stimuli were presented far enough apart in time, participants were able to respond to them as if they were responding to two sequentially presented unimodal stimuli, thus preventing the occurrence of the Colavita effect. In sum, therefore, the results of Experiment 5.1 suggest that the extent to which participants perceive the auditory and visual stimuli as constituting a unitary multisensory (i.e., audiovisual) 156

CHAPTER 5 - AUDIOVISUAL ASYNCHRONY

event (cf. Spence, 2007) may not play such a critical role in the occurrence of the Colavita effect. According to the unity effect, the stronger an observer’s assumption that the two events refer to the same unimodal object rather than to two separate events, the greater the intersensory bias, or intersensory binding, between them (see Spence, 2007, for a recent review). The extent to which participants perceive stimuli as constituting a unitary event depends on factors such as the temporal and spatial coincidence of the stimuli (e.g., Lewald, Ehrenstein, & Guski, 2001; Slutsky & Recanzone, 2001; Spence, 2007; Stein & Meredith, 1993; Vatakis & Spence, 2007b; Wallace, Roberson, Hairston, Stein, Vaughan, & Schirillo, 2004; Welch, 1999; Welch & Warren, 1986; Zampini, Guest, Shore, & Spence, 2005a). Thus, the finding that the occurrence of the Colavita effect does not appear to depend on the extent to which participants temporally bind the auditory and visual stimuli into a single audiovisual percept (or event) is inconsistent with the notion that the unity effect contributes to it. Having investigated whether temporal factors contribute to the Colavita effect, the aim of Chapter 6 was to investigate whether spatial factors can also modulate the Colavita effect. The main aim of the experiment reported in Chapter 6 was to investigate whether the spatial separation between the auditory and visual stimuli can contribute to the magnitude of the Colavita effect. The design of the experiment was similar to that of Experiment 2.1, except that the relative spatial positions of the auditory and visual components of the bimodal stimuli were modulated (they could be presented from the same or different side of space).

157

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

CHAPTER 6
6.0 DOES SPATIAL COINCIDENCE MODULATE THE COLAVITA EFFECT?
Spatial coincidence has been shown to modulate multisensory integration under many conditions, with the likelihood of multisensory binding increasing when the stimuli come from the same rather than from different spatial positions (see Spence & Driver, 2004; Stein & Meredith, 1993, for reviews). For example, spatial coincidence has been shown to modulate the integration of audiovisual motion signals (e.g., Hall & Earle, 1954; Meyer, Wuerger, Röhrbein, & Setsche, 2005; Soto-Faraco et al., 2002), as well as judgments of the temporal order, or simultaneity, of stimuli (Zampini et al., 2003a, 2003b, 2005a). However, many other audiovisual interactions have been shown to be completely unaffected by whether the auditory and visual stimuli are presented from the same position or not: For example, the well-known McGurk effect (Bertelson, Vroomen, Wiegeraad, & de Gelder, 1994; Colin, Radeau, Deltenre, & Morais, 2001; Fisher & Pylyshyn, 1994; Jones & Munhall, 1997), the phenomenon of auditory driving (Recanzone, 2003; Regan & Spekreijse, 1977), and temporal ventriloquism (Vroomen & Keetels, 2006). The audiovisual interactions which are affected by the spatial coincidence of the stimuli tend to be those which involve spatial judgments, while the ones which are

158

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

unaffected tend to involve only judgments concerning the identification of the stimuli. Note, though, that it is sometimes hard to distinguish tasks involving spatial judgments from tasks which do not. While the Colavita effect could be thought of as an identification task, where participants have to determine the modality of the target, it could also be thought of as a detection task. Hence, it is as yet unclear as to whether spatial factors would influence the Colavita effect. Therefore the first aim of the experiment reported in the present chapter (Experiment 6.1) was to determine whether spatial factors can modulate the magnitude of the Colavita effect. To date, just one previous study has investigated the role of spatial factors in the Colavita effect. Johnson and Shapiro (1989) presented participants with the Colavita task and varied whether the unimodal auditory, unimodal visual, and bimodal stimuli were presented ‘predictably’ (i.e., when the auditory and visual stimuli were always presented from the same position, 10° to the left of the participant) or else ‘unpredictably’ (i.e., where the auditory stimuli could be presented randomly from one of four positions equidistantly-spaced around the participant, while the visual stimuli were presented randomly on a circle of 11.5° in diameter, centred on fixation). A Colavita visual dominance effect was observed in the ‘predictable’ condition but not in the ‘unpredictable’ condition, leading Johnson and Shapiro to argue that vision may primarily tend to dominate in the Colavita effect when the stimuli come from ‘predictable’ spatial locations. It is important to note, however, that there is a potential confound in Johnson and Shapiro’s (1989) study. Namely, the stimuli in the ‘predictable’ condition always came from the same location (which was at a point to the left of fixation), whilst the stimuli in the ‘unpredictable’ condition were more likely to be presented from locations that were further apart (up to 170° apart, with the visual stimuli being

159

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

presented from around fixation while the auditory stimuli were presented from almost directly behind the participant). Thus, one could also explain Johnson and Shapiro’s results in terms of the Colavita visual dominance effect simply being more prevalent when the auditory and visual stimuli came from the same (or similar) positions than when they came from different positions (which could provide possible support for the notion that the unity effect could contribute to the Colavita effect). However, based on the evidence from Johnson and Shapiro’s study alone, one cannot be fully certain as to whether the Colavita effect is affected by the relative positions (same vs. different) from which the stimuli are presented. Another question to arise from Johnson and Shapiro’s (1989) study concerns whether stimulus eccentricity also affects the magnitude of the Colavita effect. Research has shown that visual performance is poorer in the periphery than in central vision in terms of slower response latencies (Chelazzi, Marzi, Panozzo, Pasqualini, Tassinari, & Tomazzoli, 1988; Goldring, Dorris, Corneil, Balyantyne, & Munoz, 1996; Marzi & Di Stefano, 1981; Marzi, Mancini, Metitieri, & Savazzi, 2005; Yao & Peck, 1997), poorer contrast sensitivity, and higher detection thresholds (see Kitterle, 1986, for a review). Meanwhile, auditory saccadic latencies tend to show the reverse pattern; responses are more rapid when auditory stimuli are presented in the periphery than at the midline (Goldring et al., 1996; Yao & Peck, 1997), which may be due to the more rapid localization of the auditory target at larger eccentricities. Hence, it is possible that the eccentricity at which the auditory and visual stimuli have been presented from in the studies reported so far in this thesis, as well as most of the previous studies of the Colavita effect (i.e., the stimuli were presented at 13° of visual angle or less from the midline; see Chapter 1, Section 1.4.5 for a summary of the

160

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

relative spatial positions of the stimuli) may have contributed to the magnitude of the Colavita visual dominance effect observed. Experiment 6.1 therefore had two goals: First, in order to investigate whether the eccentricity at which the stimuli are presented would contribute to the magnitude of the Colavita effect (cf. Johnson & Shapiro, 1989), participants were presented with a version of the Colavita task in which the relative spatial positions of the auditory and visual components of the bimodal stimulus were varied (where they were presented from the same or different spatial positions, and at eccentricities of 13° or 26° from the midline). If the Colavita effect observed in the experiments reported in this thesis (where the auditory and visual stimuli have always been presented from straight ahead, at fixation) has been caused (at least in part) by a crossmodal bias attributable to the dominance of stimuli that are presented at fixation, one might expect to see a diminished (or possibly even) reversed Colavita effect when auditory and visual stimuli are presented more peripherally. The second goal of Experiment 6.1 was to investigate whether the Colavita effect is modulated by the extent to which participants spatially bind the auditory and visual stimuli. If the Colavita effect is modulated by the extent to which participants perceive the auditory and visual stimuli as constituting a single audiovisual event, then the emergence of the Colavita effect should depend on whether the auditory and visual stimuli come from the same spatial position or from different spatial positions. The design of the experiment was similar to that reported in Experiment 2.1, except that the relative spatial positions and eccentricities of the auditory and visual components of the bimodal stimuli were modulated (they could be presented from the same or different side of space, and could be presented at 13° or 26° from the midline).

161

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

6.1

EXPERIMENT 6.1

6.1.1 Methods Participants. 36 naïve participants (mean age of 23 years, ranging from 18-34 years; 15 males and 21 females) took part in Experiment 6.1. All except two of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes. Apparatus and materials. These were exactly the same as in Experiment 2.1 with the exceptions that the auditory and visual stimuli were presented at an eccentricity of 13° or 26° of visual angle horizontally from the midline (18 participants in each condition; the 13° and 26° Stimulus Eccentricity conditions), and could be presented from either the same side or from different sides (during bimodal trials). As in Experiment 2.1, the auditory stimuli were presented from a loudspeaker cone situated 2cm directly behind the visual light sources. Hence, on those bimodal trials in which the auditory and visual stimuli were presented from the same side they appeared from exactly the same spatial position. Finally, the last difference was that the auditory stimulus consisted of a 50ms burst of white noise which had amplitude enveloping applied to the first and last 5ms. Design. The design was exactly the same as in Experiment 2.1 with the exceptions that participants initially completed two blocks of 12 trials in which they had to specify whether the stimulus (a light in one block and a sound in the other block) had been presented from the left or right. In both the 13° and the 26° Stimulus Eccentricity conditions, participants made less than 1% errors, thus demonstrating that they were easily able to discriminate between the two target locations. Next, the participants were presented with 6 blocks of 100 trials (where each block consisted of 40 visual, 40 auditory, and 20 bimodal trials). The unimodal targets were presented 162

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

equally often from either side. For the bimodal target trials, the four possible combinations (auditory left-visual left, auditory left-visual right etc) were also presented equally often. Procedure. This was the same as in Experiment 2.1 with the exception that the ISI ranged randomly (with a rectangular distribution) between 1450ms and 1700ms.

6.1.2 Results Error data. The participants failed to respond on 4.7% of the trials overall, and these trials were not included in the data analyses. Preliminary analyses of the data revealed no significant main effect of the side on which the targets were presented (i.e., left vs. right), and so the data were combined across this factor to simplify the data analysis. The RT and error data from Experiment 6.1 are shown in Figures 6.1 and 6.2, and in Table 6.1.

*
8 7 6 5 4 3 2 1 0 Same position Different position Same position Different position

* ** * ***

* *

Errors (%)

Auditory-only Visual-only

13° Stimulus eccentricity

26° Stimulus eccentricity

* ** ***

p < .05 p < .01 p < .001

Figure 6.1. Figure showing the percentages of auditory-only and visual-only errors when the auditory and visual stimuli were presented from the same or different positions in the 13° and 26° stimulus eccentricity conditions in Experiment 6.1. The error bars indicate the standard errors of the means.

163

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

In order to investigate what effect, if any, the spatial separation and/or relative spatial position of the auditory and visual stimuli had on the magnitude of the Colavita effect, an ANOVA was conducted on the data from the bimodal trials in which the participants failed to respond correctly. The within-participants factors were Response (Auditory-only or Visual-only) and Spatial Position (Same or Different), and the between-participants factor was Stimulus Eccentricity (13° or 26°). The analysis revealed a significant main effect of Response [F(1, 34) = 30.11, p < .001] overall, as expected, with participants making significantly more visual-only than auditory-only responses (4.1% vs. 1.6% of all bimodal trials), thus demonstrating a robust Colavita effect. The analysis also revealed a significant main effect of Spatial Position [F(1, 34) = 23.05, p < .001], attributable to the fact that participants made more errors when the auditory and visual stimuli were presented from the same position (3.6% errors) than when they were presented from different positions (2.1% errors).
Table 6.1. Mean error rates for unimodal auditory, unimodal visual, and bimodal target stimuli. Mean reaction times (RTs) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli. The stimuli were presented at an eccentricity of either 13° or 26° (varied between participants), and from either the same position or from different positions. Standard errors are shown in parentheses.
Targets 13° eccentricity Unimodal Same position Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses 569 (23) 536 (18) 545 (31) 536 (23) 5.2 (1.1) 5.4 (1.1) 5.0 (0.8) 4.8 (0.6) Bimodal Different positions 26° eccentricity Unimodal Same position Bimodal Different positions

1.7 (0.9) 4.5 (0.9)

1.6 (0.9) 3.1 (0.7)

2.3 (0.4) 5.9 (0.8)

0.9 (0.4) 2.9 (0.8)

647 (20) 616 (22)

648 (16) 614 (19)

673 (26) 666 (25)

657 (26) 649 (27)

164

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

Importantly, the interaction between these two factors, Response and Spatial Position, was also significant [F(1, 34) = 6.97, p = .012], showing that the Colavita effect was significantly larger (t(35) = 2.66, p = .012) when the targets came from the same position (mean difference between visual-only and auditory-only errors = 3.2% errors; t(35) = 5.90, p < .001) than when they came from different positions (mean difference = 1.7%; t(35) = 3.43, p = .002; see Figure 6.1). Finally, there was a significant interaction between Spatial Position and Stimulus Eccentricity [F(1, 34) = 6.16, p = .018], attributable to the effect of Spatial Position being larger (t(17) = 2.42, p = .022) at the 26° eccentricity (mean difference between same and different positions = 2.3% errors; t(17) = 4.16, p = .036) than at the 13° eccentricity (mean difference = 0.7% errors; t(17) = 2.45, p = .025). None of the other terms in the analysis reached significance: for Stimulus Eccentricity, or for the interactions between Response × Stimulus Eccentricity, or Response × Spatial Position × Stimulus Eccentricity, for all the terms [F < 1, n.s.]. An ANOVA performed on the error data from both the unimodal and bimodal trials with the factors of Stimulus (Auditory, Bimodal, or Visual) and Stimulus Eccentricity (13° or 26°), revealed a significant main effect of Stimulus [F(1.13, 30.51) = 18.59, p < .001]. This was attributable to participants making significantly more errors on the bimodal trials (11.4% errors) than on either the unimodal auditory (5.1% errors; t(35) = 4.50, p < .001) or the unimodal visual trials (5.1% errors; t(35) = 4.41, p < .001), but made no more errors on unimodal visual than on unimodal auditory trials (t(35) = .09, p = .932). None of the other terms reached significance: for Stimulus Eccentricity, or for the interaction between Stimulus × Stimulus Eccentricity, for both terms [F < 1, n.s.].

165

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

RT data. The RT data from the bimodal trials in which participants responded correctly were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), Spatial Position (Same or Different), and Stimulus Eccentricity (13° or 26°; see Figure 6.2). This analysis revealed a significant main effect of Target Modality [F(1, 34) = 10.75, p = .002], with participants responding more rapidly to visual stimuli (636ms) than to auditory stimuli (657ms), overall. There was also a significant interaction between Target Modality and Stimulus Eccentricity [F(1, 34) = 4.20, p = .048], attributable to the effect of Target Modality being significant at the 13° but just failing to reach significance at the 26° Stimulus Eccentricity condition (mean difference between visual and auditory responses = 24ms vs. 19ms, respectively; t(17) = 4.43, p < .001; t(17) = 1.18, p = .089). None of the other terms reached significance: for Spatial Position [F(1, 34) = 2.88, p = .100], Stimulus Eccentricity, Target Modality × Stimulus Eccentricity, Target Modality × Spatial Position × Stimulus Eccentricity, for these terms [F < 1, n.s.].

720 700 680 660
RTs (ms)

***

***

640 620 600 580 560 540 Same position Different position Same position Different position Auditory targets Visual targets

13° Stimulus eccentricity

26° Stimulus eccentricity

***

p < .001

Figure 6.2. Figure showing the mean RTs to auditory and visual targets (on bimodal trials) when they were presented from the same or different positions in the 13° and 26° stimulus eccentricity conditions in Experiment 6.1. The error bars indicate the standard errors of the means.

166

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

Finally, the unimodal and bimodal RT data were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), Target Type (Unimodal or Bimodal), and Stimulus Eccentricity (13° or 26°). This analysis revealed a significant main effect of Target Modality [F(1, 34) = 19.12, p < .001], with participants once again responding more rapidly to visual stimuli (591ms) than to auditory stimuli (614ms). There was also a significant main effect of Target Type [F(1, 34) = 251.14, p < .001], attributable to the fact that participants responded more rapidly to the unimodal (559ms) than to the bimodal targets (647ms). None of the other terms reached significance: for Stimulus Eccentricity, or for the interactions between Stimulus Eccentricity × Target Type, and Stimulus Eccentricity × Target Modality × Target Type, for the three terms [F < 1, n.s.], or for the interactions between Target Modality × Stimulus Eccentricity [F(1, 34) = 3.47, p = .071], Target Type × Stimulus Eccentricity [F(1, 34) = 2.70, p = .109].

6.1.3 Discussion The results reported in Experiment 6.1 demonstrate a significant Colavita effect; that is, when participants failed to respond correctly on the bimodal target trials (which they did on 5.7% of all bimodal trials), they made significantly more visualonly than auditory-only responses (4.1% vs. 1.6% of all bimodal trials, respectively). The magnitude of the Colavita effect (the percentage of visual-only response minus the percentage of auditory-only responses) was found to be significantly larger when the auditory and visual stimuli were presented from the same position (3.2%) than when they were presented from different positions (1.7%)1. Though, note that the

1

Note that the larger Colavita effect observed when the stimuli were presented from the same position was specifically attributable to participants making more visual-only responses in the same position

167

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

Colavita effect was significant in both the same position and different position positions. By contrast, the size of the Colavita effect itself was not significantly affected by the magnitude of the spatial separation (or eccentricity) between the stimuli (mean Colavita effect of 2.1% at the 26° separation, and 2.8% at the 52° separation). Hence, the target eccentricity did not exert any significant effect on the magnitude of the Colavita effect (at least for the particular stimulus eccentricities tested here).

6.2

GENERAL DISCUSSION
The main finding to emerge from Experiment 6.1 was that the magnitude of

the Colavita effect was significantly larger when the auditory and visual stimuli were presented from the same position than when they were presented from different positions (mean magnitude of the Colavita effect: 3.2% vs. 1.7%, respectively), although it is important to note that the Colavita effect was significant whether or not the auditory and visual stimuli were presented from the same or different positions. It should also be noted here that these results support the argument outlined earlier that the larger Colavita effect observed when the auditory and visual stimuli came from ‘predictable’ than from ‘unpredictable’ locations in Johnson and Shapiro’s (1989) previous study might be explained by the spatial coincidence between the auditory and visual stimuli, rather than by the ‘predictability’ of the stimulus locations per se. The finding that the spatial coincidence (whether the stimuli were presented from the same or different spatial positions) between the auditory and visual stimuli

condition, rather than due to a general increase in the error rate in the same position condition overall; Participants made more visual-only (t(17) = 4.02, p < .001), but no more auditory-only (t(17) = 1.49, p = .149), responses when targets were presented from the same rather than different locations.

168

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

can modulate the magnitude of the Colavita effect could be taken to support the notion that the extent to which participants spatially bind the auditory and visual stimuli contributes to the Colavita effect. On the other hand, however, the fact that the Colavita effect was significant even when the auditory and visual stimuli were presented from different positions suggests that the extent to which participants spatially bind the stimuli may not play such a crucial role in the emergence of the Colavita effect (otherwise one might expect the effect to be totally eliminated). Indeed, another factor, apart from spatial binding, could also plausibly explain why the Colavita effect was smaller when the auditory and visual stimuli were presented from different spatial locations versus the same location; participants could use redundant spatial cues to improve their performance on trials in which the stimuli were presented from different locations. The results of crossmodal TOJ studies have shown that the accuracy with which participants can judge the temporal order of an auditory and a visual stimulus improves when the stimuli come from different spatial locations rather than from the same spatial location (Bertelson & Aschersleben, 2003; Keetels & Vroomen, 2005; Spence, Baddeley, Zampini, James, & Shore, 2003; Zampini, Guest, & Shore, 2005a; Zampini, Shore, & Spence, 2003b). The main account that has been put forward (see Keetels & Vroomen, 2005; Spence et al., 2003, for reviews) to explain this pattern of results is that when the auditory and visual stimuli are spatially separated, participants may use the redundant spatial information available to facilitate their performance; participants can combine the information relating to which side the first stimulus appears on, and the information regarding the relative spatial positions of the stimulus modalities, to infer which modality was presented first. As participants have been clearly demonstrated to exploit redundant spatial cues in crossmodal TOJ tasks

169

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

(especially at large spatial separations such as those used in Experiment 6.1), participants in the Colavita task might also use redundant spatial cues to inform their decisions. For example, when the auditory and visual stimuli were presented from different spatial locations, participants could use the redundant spatial cues informing them that one stimulus had been presented at each of the spatial locations to infer that a bimodal stimulus had been presented. Therefore, although the results of Chapter 6 could be taken to support the notion that the extent to which participants spatially bind the auditory and visual stimuli modulates the Colavita effect, these results could also be taken to suggest that the Colavita effect is attenuated when participants have a redundant spatial cue informing them that two stimuli had been presented (rather than just one stimulus). In sum, the results of Chapters 5 and 6 have revealed that both the temporal and spatial separation between the auditory and visual components of a bimodal stimulus can modulate the Colavita effect. However, the results do not necessarily support the notion that the assumption of unity between the auditory and visual stimuli on bimodal trials (i.e., the extent to which participants temporally or spatially bind the auditory and visual stimuli and perceive them as a unitary audiovisual event; Spence, 2007; Vatakis & Spence, 2007b) modulates the Colavita effect. The results of Chapter 5 can be explained in terms of the Colavita effect being attenuated or eliminated when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli, while the results of Chapter 6 could be taken to support the notion that the Colavita effect is attenuated when participants have a redundant spatial cue informing them that two stimuli had been presented.

170

CHAPTER 6 - SPATIAL COINCIDENCE & THE COLAVITA EFFECT

While a great deal of research has demonstrated that audiovisual integration is modulated by the structural properties (i.e., those properties relating to the physical properties of the stimuli) of the audiovisual inputs (i.e., their spatial separation and temporal synchrony; see Vatakis & Spence, 2007b, for a review), another factor that has recently been shown to modulate audiovisual integration is the semantic congruency between the auditory and visual stimuli (i.e., whether or not the stimuli depict the same object or event; Laurienti et al., 2004; Molholm et al., 2004; Taylor et al., 2006; Vatakis & Spence, 2007b). The aim of the experiments reported in Chapter 7 was to determine whether the semantic congruency between the auditory and visual stimuli can contribute to the Colavita effect. The design of the experiments in Chapter 7 were similar to that of Experiment 2.1, except that the semantic congruency between the auditory and visual stimuli was modulated: the auditory and visual stimuli could depict the same animal (e.g., the sound and picture of a cat2), or different animals (e.g., the sound of a cat, and a picture of a dog).

Note that the stimuli used in the experiments reported in Chapter 7 were pictures and sounds of animals, rather than the simple light and tone stimuli used in the previous experiments reported in this thesis.

2

171

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

CHAPTER 7
7.0 DOES SEMANTIC CONGRUENCE MODULATE THE COLAVITA EFFECT?
Although the experiments reported in Chapters 5 and 6 revealed that the Colavita effect is influenced by the temporal and spatial separation of the auditory and visual stimuli on bimodal trials, they do not necessarily support the notion that the Colavita effect is modulated by the extent to which participants perceive the auditory and visual components (of the bimodal stimulus) as a unitary audiovisual event (Spence, 2007; i.e., the unity effect). It is possible, however, that the Colavita visual dominance effect might be modulated by factors other than spatial and temporal coincidence that have also been shown to modulate the strength of the participants’ assumption of unity. Two such factors that could be considered in this context would be stimulus congruency and semantic congruency; where the term stimulus congruency applies to simple stimuli which are congruent on dimensions (such as brightness, volume, etc), and the term semantic congruency applies to meaningful stimuli (e.g., pictures and sounds of real world objects) which are congruent in terms of the object or event they depict. Multisensory cues that originate from a single object (or event) will typically share not only their temporal and spatial attributes, but may also share certain

172

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

semantic features learnt through prior experience (Laurienti et al., 2004), or the same linguistic labels (e.g., a ‘low’ pitched tone or a ‘low’ spatial position; see Marks, 1982; Mudd, 1963). Thus, the stimulus congruency and/or semantic congruency between stimuli originating from different sensory modalities may also help to facilitate the crossmodal binding of sensory information. Indeed, it is plausible to suggest that the stimulus or semantic congruency between stimuli should, in fact, enrich any assumption of unity concerning them. However, the majority of the multisensory studies that have been published to date investigating the Colavita effect have tended to use simple stimuli with little (if any) semantic content (e.g., bursts of noise or flashes of light have been used particularly frequently; see Sinnett et al., 2007, on this point) and the effects of stimulus or semantic congruency on the Colavita effect have never received any attention by researchers thus far. In studies of crossmodal speeded classification, it is usually found that an attribute of a stimulus in one modality will affect a participant’s categorisation of an attribute of a stimulus in another modality (e.g., see Marks, 2003, for a review). In the typical crossmodal speeded classification paradigm, participants have to make speeded classification responses to stimuli in one modality according to values on one dimension (e.g., visual brightness) while values on another (irrelevant) dimension of a stimulus presented in another modality (e.g., the volume of a simultaneouslypresented auditory stimulus) vary orthogonally. It is typically found that participants’ classification responses are faster when the stimuli are congruent (e.g., a bright light paired with a loud tone rather than a low pitched tone; Marks, 1987) than when they are incongruent (e.g., a low-pitched tone presented at a high spatial position rather than at a low spatial position; Ben-Artzi & Marks, 1995). Thus, the finding that the congruency between auditory and visual stimuli appears to facilitate the classification

173

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

of the stimuli suggests that the stimulus congruency between stimuli could potentially strengthen any assumption of unity that a participant may have concerning them. In order to test whether the stimulus congruency between the auditory and visual stimuli would affect the magnitude of the Colavita effect, one would have to present the participant with auditory and visual stimuli varying in a combination of dimensions which yield stimulus congruency effects. The studies of crossmodal classification have revealed that effects of stimulus congruency occur between audition and vision for the following combinations of auditory and visual dimensions; pitch and visual position (Ben-Artzi & Marks, 1995; Melara & O’Brien, 1987; Patching & Quinlan, 2002), pitch and visual lightness (Marks, 1987; Martino & Marks, 1999), pitch and brightness (Marks, 1987), loudness and brightness (Marks, 1987), and pitch and visual size (Gallace & Spence, 2006). As all of these stimulus combinations which would yield stimulus congruency effects would require modulating the relative intensities or sizes of the auditory and visual stimuli, or their relative spatial positions (factors which could potentially modulate the magnitude of the Colavita effect), it was not viable to investigate the effects of stimulus congruency on the Colavita effect. A few recent studies that have examined the role of semantic congruency on multisensory integration have shown that the behavioural aspects of audiovisual object recognition can also be affected by the semantic congruence between the component unisensory stimuli under certain conditions. In particular, participants tend to respond more rapidly and accurately to semantically congruent auditory and visual stimuli than to semantically incongruent stimulus pairings (e.g., Laurienti et al., 2004; Molholm et al., 2004) in tasks where participants have to respond to a pre-specified target. It is important to note, however, that while certain studies have shown the

174

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

speed and accuracy of object recognition to be affected by audiovisual semantic congruence, other studies have failed to demonstrate any such effects on human behaviour. For example, Taylor et al. (2006) found that the semantic congruency between auditory and visual stimuli on bimodal trials modulated activity in the perirhinal cortex but had no main effect on the accuracy or RTs of the participants’ responses. Furthermore, other researchers have tended to focus solely on the effect of semantic congruency on brain activity without even analyzing the behavioural data (e.g., Beauchamp, Lee, Argall, & Martin, 2004). To date, only one study has attempted to extend the Colavita effect to the processing of more complex and meaningful stimuli. In particular, Sinnett et al. (2007; also see Chapter 4, Introduction, for a description) conducted a Colavita study using complex sounds (such as a cat meowing) and line drawings of familiar objects (such as a light bulb) as stimuli. In Sinnett et al.’s study, the participants had to monitor the stimulus stream for predefined auditory, visual, or bimodal targets (i.e., the sound of a cat meowing, a picture of a stoplight, or both stimuli presented simultaneously) amongst a continuous stream of irrelevant distractors. However, while Sinnett and his colleagues demonstrated significant Colavita effects in the experiments in their study whilst using these meaningful stimuli, they made no attempt to investigate the influence of semantic congruency on the magnitude of any Colavita effects reported. That is, they did not address the question of whether or not the Colavita effect would be larger for semantically congruent audiovisual pairings than for semantically incongruent pairings of stimuli. Instead, the stimulus pairings that were presented on the bimodal trials were always semantically incongruent (e.g., a picture of traffic lights paired with the sound of a bird). Hence, although the Colavita effect has been demonstrated using complex stimuli, no attempt has as yet

175

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

been made to investigate the effect of semantic congruency on the magnitude of the visual dominance effect. The main aim of the experiments reported in this chapter was therefore to investigate whether the Colavita effect would be modulated by the semantic congruency between the auditory and visual stimuli. The design of the experiments in Chapter 7 was similar to that reported in Experiment 2.1, with the exception that the semantic congruency between the auditory and visual stimuli was modulated. In Experiment 7.1, the participants were presented with auditory and visual stimuli from two different classes of animal (cats and dogs), as well as bimodal stimuli which could either be composed of congruent or incongruent pairings of audiovisual stimuli. One might expect the Colavita effect to be larger in these experiments (i.e., when using complex, meaningful stimuli) than in the previous experiments reported in this thesis (in which simple stimuli were presented), given that the presentation of more complex stimuli should presumably increase the perceptual load of the participants’ task somewhat (e.g., Lavie, 2005), which would be expected to increase the overall error rate.

7.1

EXPERIMENT 7.1

7.1.1 Methods Participants. 12 naïve participants (mean age of 21 years, age range from 1828 years; 6 males and 6 females) took part in Experiment 7.1. All except two of the participants were right-handed by self-report. The experimental session lasted for approximately 25 minutes.

176

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

Apparatus and materials. As in Experiment 2.1, the participants sat approximately 60cm from the light and sound sources in a dimly-illuminated testing booth. A 17” FD Trinitron CRT monitor (60Hz refresh rate) positioned directly in front of the participant at eye-level was used to present the visual stimuli. The visual stimuli consisted of full-colour photographs (15cm × 25cm) of either a cat or dog, subtending approximately 8° of visual angle on a white background for 350ms. The auditory stimuli (the sound of a cat meowing or a dog barking) were presented for 350ms from two loudspeakers; one positioned 24cm to either side of the centre of the monitor, such that the auditory and visual stimuli appeared to emanate from the same position. As in Experiment 2.1, the sounds were presented at 65dB(A), as measured from the participant’s ear position and amplitude enveloping was applied to the first and last 5ms of the auditory stimulus. There was one exemplar of each type of sound or image. There were four possible unimodal stimuli which could be presented together in any combination to give rise to four possible combinations of bimodal stimuli, consisting of both semantically congruent and semantically incongruent stimulus pairings (see Table 7.1). Finally, participants responses were collected, and they were instructed to respond, in exactly the same way as in Experiment 2.1.
Table 7.1. Table showing the semantic category (cat vs. dog), semantic congruency (congruent vs. incongruent), and the number of auditory and visual stimuli presented in each block in Experiment 7.1. Semantic category Auditory Visual stimulus stimulus Cat Dog Cat Dog Cat Cat Dog Dog Cat Dog Cat Dog

Target type Unimodal

Semantic congruency Congruent Incongruent Incongruent Congruent

Number of trials per block (100 trials in total) 20 20 20 20 5 5 5 5

Bimodal

177

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

Design. The participants initially completed two blocks of 12 trials in which they had to specify whether the stimulus (an auditory stimulus in one block and a visual stimulus in the other block) represented a cat or a dog. None of the participants made any errors on these trials. Next, the participants were presented with 6 blocks of 100 trials (where each block consisted of 40 visual, 40 auditory, and 20 bimodal trials; just as in Experiment 2.1). See Table 7.1 for the relative frequencies of presentation of the four types of unimodal stimuli and the four types of bimodal stimuli. Procedure. This was the same as in Experiment 2.1 with the exception that the ISI ranged randomly (with a rectangular distribution) between 1450ms and 1700ms.

7.1.2 Results
Table 7.2. Mean error rates for unimodal auditory, unimodal visual, and bimodal targets in Experiment 7.1. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli (congruent or incongruent). Standard errors are shown in parentheses. Unimodal Bimodal Congruent Incongruent

Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses RTs (ms) Unimodal auditory Unimodal visual Bimodal Auditory responses Visual responses

8.6 (2.1) 11.4 (2.3)

-

-

-

7.1 (1.7) 19.9 (4.5)

5.4 (2.3) 16.4 (3.9)

577 (30) 522 (30)

-

-

-

607 (24) 602 (26)

574 (26) 579 (29)

Error data. The participants failed to respond on 0.6% of the trials overall, and these trials were not included in the data analyses. The RT and error data from 178

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

Experiment 7.1 are shown in Table 7.2 and Figure 7.1. Preliminary analyses of the data revealed no significant main effect of the identity of the targets (i.e., cats vs. dogs), and so the data were combined across this factor to simplify the data analysis.

30

*
25

**

20

Errors (%)

15

Auditory-only Visual-only

10

5

0 Congruent Incongruent

* **

p < .05 p < .01

Figure 7.1. Figure showing the percentages of auditory-only and visual-only errors made by participants when the auditory and visual stimuli depicted congruent or incongruent animals in Experiment 7.1. The error bars indicate the standard errors of the means.

The data from the bimodal trials in which the participants failed to respond to one of the two stimuli were analysed using an ANOVA with the factors of Response (Auditory-only or Visual-only) and Semantic Congruency (Congruent or

Incongruent). The analysis of the error data revealed a significant main effect of Response [F(1, 11) = 10.34, p = .008], with participants making significantly more visual-only than auditory-only responses (18.2% vs. 6.2% of all bimodal trials, respectively); demonstrating a robust Colavita visual dominance effect. The analysis of the error data also revealed a significant main effect of Semantic Congruency [F(1, 179

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

11) = 6.91, p = .023], attributable to participants making significantly more errors when they were presented with a semantically congruent bimodal stimulus (13.5% errors) than when presented with a semantically incongruent one (10.9% errors). Crucially, however, there was no significant interaction between Response and Semantic Congruency [F < 1, n.s.], thus suggesting that the magnitude of the Colavita effect itself was not modulated by the semantic congruency between the auditory and visual components of the bimodal stimulus (see Figure 7.1). Next, the congruent and incongruent bimodal target data were combined in order to compare the unimodal with the bimodal error data. An ANOVA revealed a significant main effect of Stimulus (Auditory, Bimodal, or Visual) [F(1.06, 11.62) = 9.13, p = .010], attributable to participants making significantly more errors on the bimodal trials (24.4% errors) than on either the unimodal auditory (8.6% errors; t(11) = 3.15, p = .009) or unimodal visual trials (11.4% errors; t(11) = 2.87, p = .015), and more errors on unimodal visual trials than on the unimodal auditory trials (t(11) = 2.71, p = .020). RT data. The RT data from those trials in which the participants responded correctly were analysed in an ANOVA with the factors of Target Modality (Auditory or Visual) and Target Type (Unimodal, Congruent Bimodal, or Incongruent Bimodal). The analysis revealed a significant main effect of Target Modality [F(1, 11) = 5.64, p = .037], with the participants responding more rapidly to visual (567ms) than to auditory targets (586ms) overall. The analysis of the RT data also revealed a significant main effect of Target Type [F(2, 22) = 36.97, p < .001], with participants responding more rapidly to unimodal (549ms) than to either congruent or incongruent bimodal targets (604ms and 576ms, respectively; t(11) = 7.34, p < .001; t(11) = 4.38, p = .001), and more rapidly to incongruent bimodal than to congruent bimodal targets

180

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

(t(11) = 5.27, p < .001). There was also a significant interaction between Target Modality and Target Type [F(1.27, 13.98) = 7.79, p = .011], attributable to there being a significant effect of Target Modality for the unimodal targets but not for either the congruent or the incongruent bimodal targets (mean difference between auditory and visual response latencies = 55ms, 5ms, and 5ms, respectively; t(11) = 3.08, p = .010; t(11) = .64, p = .530; t(11) = .63, p = .542, respectively). Note that there was a speed accuracy trade-off in participants’ responses to the unimodal auditory and unimodal visual targets.

7.1.3 Discussion The results of Experiment 7.1 revealed a robust Colavita visual dominance effect; that is, when participants failed to respond correctly on the bimodal target trials (which they did on 24.4% of all bimodal trials), they made significantly more visualonly than auditory-only responses (18.2% vs. 6.2% of all bimodal trials, respectively). These findings therefore demonstrate that the Colavita effect can be extended to the processing of more complex stimuli, in addition to the simple lights and sounds that have been used in the majority of previous research (see also Sinnett et al., 2007). Note that the magnitude of the Colavita visual dominance effect (the percentage of visual-only responses minus the percentage of auditory-only responses) in Experiment 7.1 using congruent (12.8%) and incongruent (11.0%) complex stimuli (color photographs and animal vocalizations) was numerically larger than that reported in many of the other experiments reported in this thesis (e.g., Colavita effect = 7.7%, 6.4%, and 4.1% in Experiments 2.1, 2.2, and 3.1). This difference may reflect the increased perceptual load attributable to the use of more complex (and thus more

181

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

attention-demanding) auditory and visual stimuli (see Lavie, 2005; Sinnett et al., 2007), resulting in a larger percentage of errors overall. The magnitude of the Colavita visual dominance effect was not affected by the semantic congruency between the auditory and visual components of the bimodal target stimuli. The comparison yielded a p value1 of .552, with a correspondingly low observed power2 of .129, which suggests that one would be justified in accepting the null hypothesis (see Frick, 1995, for the criteria that should be met before one accepts the null hypothesis) that the Colavita visual dominance effect is simply not modulated by the semantic congruency between the auditory and visual stimuli on bimodal target trials. It is important to note, however, that semantic congruency did influence performance. In particular, semantic congruency influenced the difficulty of processing of the bimodal stimuli; with significantly slower RTs and higher error rates being reported for congruent bimodal targets (604ms; 13.5% errors) than for incongruent bimodal targets (576ms; 10.9% errors). The slower RTs and higher error rates observed for congruent bimodal targets (than incongruent targets) suggest that participants found it harder to separate the auditory and visual components of the congruent stimuli (i.e., it took them longer to realise that a bimodal stimulus had been presented) than to separate the components of the incongruent stimuli. This would have resulted in participants taking longer to respond to both components of the

Frick (1995) has argued that one cannot confidently accept the null hypothesis when the p value is in the range from .200 to .500, but if the p value is greater than .500, he argues that this provides one important criterion for accepting the null hypothesis. 2 Observed power is the probability of correctly rejecting a false statistical null hypothesis (Type II error; the probability of a Type II error is referred to as β) and is equal to 1 − β. Thus, while a low pvalue and a high observed power would provide support for the H1 hypothesis, a p-value above .500 and a low observed power would be evidence supporting the null hypothesis (Frick, 1995).

1

182

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

bimodal target and making more errors on the bimodal trials (i.e., they responded to only one component of that stimulus). One factor that may have contributed to participants responding more rapidly and accurately on the incongruent bimodal target trials (as opposed to on the congruent target trials) may have been that the mismatch between the stimuli for the incongruent bimodal targets could have provided participants with an extra cue to inform them that a bimodal target had, in fact, been presented. Another explanation for participants’ difficulty in responding to congruent (vs. incongruent) bimodal targets could be a ‘failure of binding’ (Baylis et al., 2002). According to this view, the auditory stimulus would fail to reach awareness because it was eclipsed by the visual information, which adequately described the auditory perception (i.e., the auditory percept is redundant). If participants had experienced such a failure to represent both components of the bimodal stimulus, it would have been more difficult for them to realise that two stimuli had been presented (which would in turn be expected to increase their RTs and error rates). In sum, the semantic congruency between the auditory and visual components of the bimodal targets influenced participants’ performance on the Colavita task (in terms of their RTs and error rates on bimodal trials). It did not, however, affect the magnitude of the Colavita effect that was observed. It has been argued elsewhere that animals have both more features per se, and also more features in common, than non-living objects (e.g., McRae, de Sa, Seidenberg, 1997; Tyler & Moss, 2001). This makes it more difficult to distinguish two animals from each other, than to distinguish an animal from a non-living object. It is therefore possible, if rather unlikely, that one reason why no effect of semantic congruency on the Colavita effect was observed in Experiment 7.1 may have been

183

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

because of the large overlap in the number of object features for the particular stimuli used. In order to address this possibility, a control study was run (using an experimental procedure and design that was identical to that used in Experiment 7.1) in which the participants (N = 14) were now presented with stimuli in different semantic categories (animals and non-living objects; e.g., cats and phones). However, once again, no significant main effect of semantic congruency on the magnitude of the Colavita effect was found (mean Colavita effect of 7.2% and 5.9% in the semantically congruent and semantically incongruent conditions, respectively). This shows that the null effect of semantic congruency on the Colavita visual dominance effect reported in Experiment 7.1 cannot simply be attributable to the specific stimuli that were presented. Another reason why no effect of semantic congruency on the Colavita effect was observed in either Experiment 7.1, or in the control experiment, may have been because there were only 4 different stimulus pairings (auditory-cat auditory-dog, auditory-cat visual-dog, etc) and only 1 exemplar of each stimulus. It may therefore have been the case that these particular stimulus pairings simply became over-learned by the participants (even for the incongruent pairings), and consequently, overrepresented in long-term episodic memory. In order to rule out this possible explanation of the null effect of semantic congruency on the magnitude of the Colavita effect reported in Experiment 7.1, the size of the stimulus set in the next experiment was increased from 4 to 80 stimuli. This modification to the design made it possible to present participants with many more different semantically congruent and semantically incongruent stimulus pairings than had been the case in Experiment 7.1. In addition, a greater number of participants were tested, thus increasing the statistical power of the experimental design, and hence making it more likely that an

184

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

effect of semantic congruency would be found, should one exist. Finally, in order to make the experiment more similar to the one reported by Sinnett et al. (2007), an extra response key was introduced (the three-key response condition; cf. Experiment 2.2). Thus, there were three separate response keys for each type of stimulus (an auditory response key, a visual response key, and a bimodal response key).

7.2

EXPERIMENT 7.2

7.2.1 Methods Participants. 30 naïve participants (mean age of 21 years, age range from 1927 years; 11 males and 19 females) took part in Experiment 7.2. All except three of the participants were right-handed by self-report. The experimental session lasted for approximately 30 minutes. Apparatus and materials. These were exactly the same as in Experiment 7.1 with the exception of the particular stimuli used and the response requirements of the task. The auditory stimuli consisted of animal sounds (8-bit; mono-channel; 11,500Hz digitization), some of which were obtained from an online library

(http://www.cofc.edu/~marcellm; downloaded 9th May 2005; for normative data concerning these stimuli, see Marcell, Borella, Greene, Kerr, & Rogers, 2000) and the rest from http://www.a1freesoundeffects.com/animal.html. The visual stimuli were comprised of 40 line-drawing pictures of different animals which were chosen from the Snodgrass and Vanderwart picture database (see Snodgrass & Vanderwart, 1980, for standardization statistics) and edited using Microsoft Paint Version 5.1. On the congruent bimodal trials, the auditory and visual stimuli depicted the same animal, while on the incongruent bimodal trials they depicted different animals. The response

185

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

requirements of the task were exactly the same as in Experiment 2.2 (i.e., the threekey response condition). Design and procedure. The design and procedure were exactly the same as in Experiment 7.1 with the exception that the participants were presented with 4 blocks of 200 trials, each consisting of 80 visual trials, 80 auditory trials, and 40 bimodal trials. The participants were presented with an equal number (20) of bimodal congruent and bimodal incongruent trials per block.

7.2.2 Results Error data. The results of Experiment 7.2 are highlighted in Table 7.3 and Figure 7.2. The data from the bimodal trials in which the participants failed to respond to one of the stimuli were analysed using an ANOVA with the factors of Response (Auditory-only or Visual-only) and Semantic Congruency (Congruent or

Incongruent). The analysis of the error data revealed a significant main effect of Response [F(1, 29) = 52.07, p < .001], attributable to participants making significantly more visual-only than auditory-only responses (24.8% vs. 8.6% of all bimodal trials, respectively); thus, a large and highly-significant Colavita visual dominance effect was once again observed. The analysis of the error data also revealed a significant main effect of Semantic Congruency [F(1, 29) = 6.61, p = .016], attributable to participants making significantly more errors on the congruent bimodal trials (17.6% errors) than on the incongruent bimodal trials (15.8% errors). Crucially, however, there was no interaction between Response and Semantic Congruency [F < 1, n.s.], as shown in Figure 7.2.

186

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

30

***

***

25

20

Errors (%)

15

Auditory-only Visual-only

10

5

0 Congruent Incongruent

***

p < .001

Figure 7.2. Figure showing the percentages of auditory-only and visual-only errors made by participants when the auditory and visual stimuli on bimodal trials were congruent or incongruent in Experiment 7.2. The error bars indicate the standard errors of the means.

An ANOVA performed on the unimodal and bimodal error data revealed a significant main effect of Stimulus (Auditory, Bimodal, or Visual) [F(1.67, 48.43) = 45.53, p < .001], attributable to participants making significantly more errors on the bimodal trials (33.4% errors) than on either unimodal auditory (7.3% errors; t(29) = 8.50, p < .001) or unimodal visual trials (14.1% errors; t(29) = 6.05, p < .001), and more errors on unimodal visual than on the unimodal auditory trials (t(29) = 3.12, p = .003). RT data. The RT data from the congruent and incongruent bimodal target trials were combined in order to compare the unimodal RT data with the bimodal RT data. An ANOVA performed on this data with the factor of Stimulus (Auditory, Bimodal, or Visual) revealed a significant main effect [F(2, 58) = 61.74, p < .001]. This term was attributable to participants responding significantly more rapidly to

187

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

unimodal visual stimuli (477ms) than to the unimodal auditory stimuli (525ms; t(29) = 5.76, p < .001) or to the bimodal stimuli (573ms; t(29) = 11.47, p < .001), and significantly more rapidly to the unimodal auditory stimuli than to the bimodal stimuli (t(29) = 5.23, p < .001).
Table 7.3. Mean error rates for unimodal auditory, unimodal visual, and bimodal targets in Experiment 7.2. Mean reaction times (RTs; ms) for correct responses to unimodal auditory, unimodal visual, and bimodal target stimuli (congruent or incongruent). Standard errors are shown in parentheses. Unimodal Bimodal Congruent Incongruent

Error rates (%) Unimodal auditory Unimodal visual Bimodal Auditory-only responses Visual-only responses RTs (ms) Unimodal auditory Unimodal visual Bimodal

7.3 (1.0) 14.1 (2.8)

-

-

9.8 (1.3) 25.4 (2.7)

7.5 (1.4) 24.2 (2.9)

525 (15) 477 (14)

567 (16)

579 (17)

Next, the congruent and incongruent bimodal target data were analysed in an ANOVA with the factor of Semantic Congruency (Congruent or Incongruent). This analysis revealed a significant main effect of Semantic Congruency [F(1, 29) = 9.78, p = .004], with the participants responding significantly more rapidly to congruent bimodal stimuli (567ms) than to incongruent bimodal stimuli (579ms). Thus, there was a speed-accuracy trade-off for responding to bimodal congruent and incongruent stimuli (i.e., participants responded more rapidly, but less accurately, to congruent than to incongruent targets).

188

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

7.2.3 Discussion The results of Experiment 7.2 revealed a robust Colavita visual dominance effect, with participants making significantly more visual-only than auditory-only responses (24.8% vs. 8.6% of all bimodal trials). In addition, no effect of semantic congruency was observed on the magnitude of the Colavita effect. The comparison yielded a p value of .664, with a correspondingly low observed power of .124, which once again supports the null hypothesis that the Colavita visual dominance effect is simply not modulated by the semantic congruency between the auditory and visual stimuli on bimodal target trials. This null effect was observed despite the fact that a greater variety of stimuli were now presented to the participants, so that there was less chance of the stimuli becoming over-learned and over-represented in the participants’ long-term episodic memory. Note that the magnitude of the Colavita effect, and the overall error rate, was numerically larger in Experiment 7.2 (Colavita effect = 16.2%, bimodal error rate = 33.4%) than in Experiment 7.1 (Colavita effect = 12.0%, bimodal error rate = 24.4%). This difference may have been due to the greater variability in the stimuli that were presented to participants, which would have been expected to increase the perceptual load of their task (Lavie, 2005), thus increasing the processing demands of the task somewhat. Indeed, Sinnett et al. (2007) recently reported a similar increase of error rates as a result of increasing the size of the stimulus set that participants were presented with. Once again, semantic congruency had a significant effect on participants’ performance (faster RTs but more errors in the congruent than in the incongruent conditions), showing that the manipulation of semantic congruency was effective in modulating certain aspects of participants’ performance. However, one question to arise from the comparison of the results of the manipulation of semantic congruency

189

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

in Experiments 7.1 and 7.2 concerns why the semantic congruency between the auditory and visual stimuli resulted in a decrease of RTs in Experiment 7.2, while apparently having the opposite effect in Experiment 7.1 (see Figure 7.3).

640 620 600 580

***

**

RT (ms)

560 540 520 500 480 Experiment 7.1 Experiment 7.2

Congruent Incongruent

** ***

p < .01 p < .001

Figure 7.3. Figure showing the mean RTs to bimodal congruent and incongruent stimuli in Experiments 7.1 and 7.2. The error bars indicate the standard errors of the means.

One explanation for the reversal of RTs for congruent and incongruent trials between Experiments 7.1 and 7.2 may be related to the different response requirements used in the two experiments (i.e., the participants responded with two responses keys in Experiment 7.1, but with three response-keys in Experiment 7.2) that makes any simple comparison of the results of the two experiments somewhat difficult. In Experiment 7.1, the participants responded to bimodal targets using the auditory and visual response keys and thus did not have to suppress their responses to the individual components of the bimodal stimuli (i.e., the participants could respond to the bimodal target in the same way as if they were responding separately to the 190

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

auditory and visual components of the stimulus). In contrast, in Experiment 7.2, participants responded to the bimodal target using a dedicated bimodal response key, which meant that on bimodal trials participants would have had to suppress their tendency to respond to the auditory and visual stimuli individually (i.e., they had to refrain from pressing either the auditory or visual response keys). In Experiment 7.2, therefore, when an incongruent bimodal stimulus was presented, the participants may have found it easier to distinguish the two stimuli (this may be reflected in the higher accuracy of their responses to the incongruent stimuli), but it may have taken them longer to respond due to their having to suppress their responses to the individual auditory and visual components of the stimulus. Whereas, when a congruent bimodal stimulus was presented, the participants may have found it harder to distinguish the two stimuli (as reflected in their higher error rate when responding to congruent stimuli), and hence may have tended to perceive the stimulus array as consisting of the presentation of a single stimulus (despite the fact that it actually consisted of stimuli in two different modalities). The participants may therefore have responded more rapidly because they did not have to suppress any tendency to respond to the auditory and visual stimuli individually.

7.2.4 Dynamic stimuli It is important to note that in the experiments reported thus far in this chapter, the auditory and visual stimuli only represented token attributes of the particular stimulus categories (e.g., cats) concerned. For example, the actual cat sounds that were presented did not directly correspond with the exact image of the cat. A further difference between the auditory and visual stimuli in Experiments 7.1 and 7.2 was that the auditory signals were dynamic (which, by definition, auditory signals must 191

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

be) whilst the visual images consisted of static images. It could therefore be argued that one reason why no modulation of the Colavita effect by the semantic congruency of the stimuli was observed in Experiments 7.1 and 7.2 was because there was no dynamic relationship between the auditory and visual stimuli3. One important question that arises here regards whether using dynamic audiovisual stimuli belonging to the same perceptual event (i.e., so that there would be a direct correspondence, rather than a token correspondence, between the auditory and visual stimuli on the congruent bimodal trials) would result in a significant modulation of the Colavita effect by the congruency of the stimuli. The time-varying correlation between auditory and visual stimuli has on occasion been shown to be a critical factor contributing to multisensory integration (Rosenblum, Wuestefeld, & Anderson, 1996); the time-varying relationship between the auditory and visual events contributes to the structure, or the physical relationship, between the auditory and visual stimuli. Therefore, a dynamic stimulus pairing would provide an even more rigorous test of the impact of congruency on the Colavita effect, because the likelihood of the auditory and visual stimuli being ‘bound’ into a singular object or event would be much greater (presumably because there would be more automaticity to the binding). For these reasons, the effects of stimulus congruency on the Colavita visual dominance effect was explored using time-varying dynamic stimuli (i.e., audiovisual speech) in a study by Koppen, Alsius, and Spence (2008, Experiment 3). The experimental procedure and design that was used was similar to that used in Experiment 7.2, with the exception that the participants (N = 15) were now presented with audiovisual dynamic speech stimuli (sound clips and video clips

It should, however, be noted that the vast majority of studies investigating the effects of semantic congruency on audiovisual integration have tended to use these types of stimulus pairings (i.e., a static visual stimulus paired with a dynamic auditory stimulus; e.g., see Adams & Janata, 2002; Beauchamp et al., 2004; Taylor et al., 2006; see Amedi et al., 2005, for a review).

3

192

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

of a speaker pronouncing the syllables ‘mo’ and ‘da’). Despite the fact that dynamic audiovisual stimuli that belonged to the same underlying perceptual event (i.e., so that there was a time-varying correspondence between the stimuli) were used (cf. Vatakis & Spence, 2007a, 2007b), no significant main effect of stimulus congruency on the magnitude of the Colavita effect was found (mean Colavita effect of 6.4% and 5.8% in the congruent and incongruent conditions, respectively). The comparison yielded a p value of .741, with a correspondingly low observed power of .061, thus, once again, supporting the null hypothesis that the Colavita visual dominance effect is not modulated by the congruency between the auditory and visual stimuli on bimodal target trials, even when dynamic stimuli from the same speech event were presented in the two modalities.

7.3

GENERAL DISCUSSION
The two experiments reported in the present chapter represent one of the very

few occasions in which the Colavita effect has been investigated using stimuli that are more complex than the simple auditory brief beeps and visual flashes that have been used in so much of the previous research on multisensory information processing (and, in particular, in previous research on the Colavita effect). The primary aim of the experiments reported in this chapter was to investigate whether manipulating the semantic congruency between the auditory and visual stimuli would influence the magnitude of the Colavita visual dominance effect. A null effect of semantic congruency on the magnitude of the Colavita effect was found in both experiments. The fact that this null result was replicated in two separate experiments fulfils the good effort criterion for accepting the null hypothesis (see Frick, 1995).

193

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

It should, however, be noted that the participants were not simply insensitive to the manipulations of semantic congruency used in the present study. Semantic congruency had a significant effect on certain aspects of participants’ behavioural performance in Experiments 7.1 and 7.2; in particular, the congruency of the bimodal targets significantly affected the speed and accuracy of participants’ response to the bimodal targets. This pattern of results suggests that participants found it harder to separate the auditory and visual components of a bimodal target when they were congruent than when they were incongruent. There was, however, no effect of semantic congruency on the magnitude of the Colavita visual dominance effect itself.

7.3.1 Semantic congruency and the redundant target effect It is interesting to contrast the null results of semantic congruency on the Colavita effect reported here with the results of previous studies where significant effects of semantic congruency on behavioural performance have been observed (Laurienti et al., 2004; Molholm et al., 2004). In the studies of both Laurienti et al. and Molholm et al., participants had to identify pre-specified targets (e.g., an auditory or visual red or blue target in Laurienti et al.’s study, or a sound or a static image of a cow in Molholm et al.’s study) which could be presented in either a congruent audiovisual pairing (e.g., the same target would be presented in both sensory modalities) or an incongruent audiovisual pairing (both a target and non-target would be presented). In both studies, the participants responded significantly faster and more accurately to targets in congruent pairings than to targets in incongruent pairings. The authors argued that the redundant target effect (Miller, 1982; the phenomenon whereby participants respond more rapidly to single versus multiple targets) may have contributed to the improved performance observed for the congruent target pairings 194

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

because participants had two targets pertaining to a single target response. In contrast, the participants in Experiments 7.1 and 7.2 had to make one response to each stimulus, thus any redundant target effects that may have been the basis of the semantic congruency effects observed in the aforementioned studies would not have affected the Colavita effects reported in this study. One final difference between Laurienti et al.’s (2004) and Molholm et al.’s (2004) studies and the experiments reported in this chapter is that they used an identification task (in which a redundant target could help to speed up the identification process), whereas participants in the experiments reported in this chapter were presented with a task in which the identification of the stimuli may not necessarily have contributed to the judgment of which modality the target appeared in. In order to investigate whether the RT benefits for semantically congruent stimuli observed in the aforementioned studies could also be observed in a Colavitatype paradigm, a follow-up study was conducted in which participants had to respond to the semantic attributes of the stimuli (cf. Laurienti et al., 2004; Molholm et al., 2004; Sinnett et al., 2007). The new participants (N = 10) were presented with the same stimuli as in Experiment 7.1, but they now had to respond to the semantic category of the target (cat or dog) rather than its modality of presentation. Thus, for the congruent bimodal trials, the participants only had to press one key (e.g., the ‘cat’ response key). Whenever an incongruent bimodal stimulus (containing both a cat and a dog target) was presented, the participants were explicitly instructed to press both response keys. Despite the fact that participants now had to respond to the semantic category of the targets, the Colavita effect was once again observed (mean Colavita effect of 10.1%; i.e., on bimodal trials the participants often failed to respond to the

195

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

semantic category which had been presented as an auditory stimulus). Furthermore, the magnitude of the Colavita visual dominance effect reported in the follow-up study (10.1%; 19.2% visual-only responses vs. 9.1% auditory-only responses) was just as large as that observed in Experiment 7.1 (10.1%; t(20) = 0.34, p = .740). Importantly, the participants in the follow-up study responded more rapidly to congruent bimodal targets (where only a single manual response was required; 512ms) than to either of the unimodal auditory (570ms) or unimodal visual targets (539ms). They also responded more rapidly to congruent than to incongruent bimodal targets (539ms vs. 712ms). This result might be explained (in the same way as the results of Laurienti et al., 2004, and Molholm et al., 2004) in terms of a redundancy gain effect (cf. Miller, 1982). Thus, it is possible to demonstrate an effect of semantic congruency, in terms of RT benefits to congruent targets, whilst using a Colavita task design in which the participants have to identify the semantic category (but not the sensory modality) of the target. However, semantic congruency does not appear to modulate the Colavita effect itself when participants have to respond to the sensory modalities of the targets. In sum, the results of Chapters 5 and 6 have revealed that both the temporal and spatial separation between the auditory and visual components of a bimodal stimulus can modulate the Colavita effect, while Chapter 7 shows that the semantic congruency between the auditory and visual stimuli does not modulate the magnitude of the Colavita effect. The results of Chapter 5 suggest that the effect of temporal separation on the magnitude of the Colavita effect can be explained in terms of the Colavita effect being attenuated or eliminated when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli (rather than being explained in terms of the Colavita effect being modulated

196

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

by the extent to which participants temporally bound the stimuli). Meanwhile, although the results of Chapter 6 could be taken to support the notion that the extent to which participants spatially bind the auditory and visual stimuli modulates the Colavita effect, the results of Chapter 6 could just as well be taken to suggest that the Colavita effect is attenuated when participants have a redundant spatial cue informing them that two stimuli had been presented (rather than just one stimulus). The assumption of unity that participants have concerning two stimuli depends on their spatial and temporal separation and can also be affected by their semantic congruency (see Vatakis & Spence, 2007b, for a review). Thus, the finding that the Colavita effect is not modulated by the semantic congruency between the stimuli nor necessarily by the extent to which participants temporally bind them (two of the factors that contribute to the unity effect) suggests that the effect of spatial separation on the magnitude of the Colavita effect (the third factor contributing to the unity effect) may be better explained in terms of the Colavita effect being attenuated when participants have a redundant spatial cue informing them that two stimuli had been presented (rather than being explained in terms of the Colavita effect being modulated by the extent to which participants spatially bind the stimuli). Thus, the results of the last three chapters do not necessarily support the notion that the assumption of unity between the auditory and visual stimuli on bimodal trials modulates the Colavita effect, rather they suggest instead that the Colavita effect is attenuated (or eliminated) when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli, or when participants have a redundant spatial cue informing them that two stimuli had been presented. While most of the experimental chapters in this thesis have focused on determining the factors that contribute to the Colavita visual dominance effect, the

197

CHAPTER 7 - SEMANTIC CONGRUENCE & THE COLAVITA EFFECT

aim of the next chapter was to explore the processes involved in the effect. It has been argued elsewhere that dissociating perceptual from post-perceptual (or more decisional) processes is an important step toward gaining a better understanding of the nature of any behavioural effect (e.g., Ashersleben, Bachmann, & Musseler, 1999; Bertelson & de Gelder, 2004; de Gelder & Bertelson, 2003). Thus, the purpose of the experiment reported in Chapter 8 was therefore to quantitatively determine the extent to which the Colavita effect reflects a perceptual and/or a decisional process.

198

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

CHAPTER 8
8.0 A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT
In crossmodal research, dissociating perceptual from post-perceptual processes represents a crucial step toward gaining a better understanding of the nature of any behavioural effect (Ashersleben et al., 1999; Bertelson & de Gelder, 2004; de Gelder & Bertelson, 2003; Watt, 1991), and of the cognitive and neural systems underlying it. Perceptual processes are those that affect the combination of multisensory cues prior to response selection/execution, whereas post-perceptual processes typically influence general response selection and/or execution mechanisms. Signal detection theory (SDT; Green & Swets, 1966; Macmillan & Creelman, 1991) provides an effective means of evaluating the relative contributions of perceptual and postperceptual effects to behaviour/perception. SDT can be used to determine the sensitivity (d’; a measure of perceptual processes which indicates how easily a target stimulus can be discriminated from the background noise) and response criterion (c; a measure of post-perceptual process which indicates how conservative/liberal participants are in terms of reporting the presence/absence of a target stimulus) that participants adopt when required to report the presence of a target stimulus. It is

199

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

therefore possible to quantitatively determine the extent to which the Colavita effect reflects a perceptual and/or a decisional process. Despite the fact that researchers have been investigating the Colavita effect for more than three decades, the relative contributions of perceptual and/or decisional processes to this visual dominance effect have never been assessed objectively before. The primary aim of the experiment reported in this chapter was therefore to assess the contributions of perceptual versus post-perceptual processes to the Colavita effect, in order to gain a better understanding of the mechanisms underlying the effect. In Experiment 8.1, the participants were first presented with psychophysical staircases in order to determine the intensity at which the auditory and visual stimuli had to be presented to the participants for the stimuli to reach the 75% correct detection threshold (the purpose of this initial part of the experiment was to derive values which could be used to present the auditory and visual stimuli at an equal level of detectability on the Colavita task). Using the stimulus intensities calculated from the detection task, the participants were next presented with a version of the Colavita task (modified to allow for SDT analysis), in which they had to detect the presence of auditory and visual targets under conditions of unimodal and bimodal stimulus presentation. If the Colavita effect represents a genuine perceptual phenomenon then one would predict that participants would show a decrease in their sensitivity to the auditory stimuli and/or an increase in their sensitivity to the visual stimuli when a bimodal stimulus was presented (compared to when they were presented unimodally). If, however, the Colavita effect is caused by post-perceptual processes instead, then one would predict participants to adopt a more conservative criterion in responding to auditory stimuli and/or a more liberal criterion for visual stimuli.

200

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

8.1

EXPERIMENT 8.1

8.1.1 Methods Participants. 22 naïve participants (mean age of 20 years, ranging from 18-25 years; 2 males and 20 females) took part in Experiment 8.1. All except three of the participants were right-handed by self-report. The experimental session lasted for approximately 45 minutes. Apparatus and materials. The stimuli were exactly the same as those used in Experiment 2.1 with the exception that the LED for the visual stimulus was white1 (whereas it was yellow in all of the previous experiments in the thesis). Each participant’s detection threshold for the auditory and visual stimuli was measured at the beginning of the experimental session, after which they completed the signal detection version of the Colavita task.

8.1.1.1 Threshold measurement task Procedure. The stimulus intensities at which participants were able to detect the auditory and visual stimuli on 75% of the trials (i.e., the 75% detection threshold) were measured in separate blocks of experimental trials. On each trial, the target was followed by a 1550ms response interval (i.e., successive stimuli were always separated by 1600ms). The participants were instructed to press the ‘b’ key whenever they detected a target, and to make no response if no target was detected. The participants were informed that the intensity of the targets would vary from trial to trial. The auditory stimuli were presented at a volume ranging between 40.2-

Presenting a white (as opposed to a yellow) LED allowed the experimenter more control over the presentation of the stimulus. It was easier to maintain the hue of a white than a yellow LED over the range of stimulus intensities presented.

1

201

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

41.8dB(A), as measured from the participant’s ear position (background noise level = 38.9dB(A)). The visual stimuli were presented at a luminance ranging between 0.640.59cd/m2. The stimuli were presented using the double-random staircase method (three-up/one-down; see Cornsweet, 1962; step size 0.1dB for the auditory stimuli, and approximately 0.03 cd/m2 for the visual stimuli). Each staircase was terminated at the twelfth reversal of the sequence; the 75% detection thresholds were calculated from the average of the last eight reversals. Design. The participants were presented with one block of trials. It took 82 (S.E. = 4) trials on average to establish the auditory threshold, and 86 trials (S.E. = 7) to establish the visual threshold. Results. The average auditory threshold was 40.8dB (A; S.E. = 0.7dB(A)), and the average visual threshold was 0.60cd/m2 (S.E. = 0.08cd/m2).

8.1.1.2 The Colavita signal detection task Procedure. An auditory target, a visual target, a bimodal target, or else no target was presented at the start of each trial, followed by a 1550ms response interval (the separation between trials was 1600ms). The targets were presented at each participant’s individually determined 75% threshold. Each trial began immediately after the end of the preceding trial. The participants were instructed to press the auditory response key when an auditory stimulus was presented, the visual response key for a visual stimulus, both response keys for a bimodal stimulus, and they were instructed not to make any response if no stimulus was presented. No specific instructions were provided to the participants as to whether they should press the two response keys simultaneously or not. The allocation of the stimuli to the auditory and visual response keys (the ‘n’ and ‘m’ keys) was counterbalanced across participants. 202

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

The participants were instructed to respond as accurately as possible (i.e., the speed of response was not stressed). No feedback regarding the correctness of the participant’s responses was provided. Design. Participants completed 3 blocks of 200 trials. Each block consisted of 50 auditory, 50 visual, 50 bimodal, and 50 target absent trials. The order of stimulus presentation was randomised within each block of trials.

8.1.2 Results Error data. The error data from the bimodal trials in which the participants failed to respond correctly were first analysed using an ANOVA with the factor of Response (Auditory-only or Visual-only). The analysis revealed a significant main effect [F(1, 21) = 7.23, p = .014], with participants making significantly more visualonly than auditory-only responses (25.4% vs. 11.2% of all bimodal trials, respectively), demonstrating a robust Colavita effect. Signal detection measures. Using SDT, one can determine the sensitivity (d’; a measure of the strength of the signal relative to the background noise) and response criterion (c; how conservative/liberal participants were in terms of reporting the presence/absence of a target stimulus) adopted by participants when they were required to report the presence of a target stimulus. The participant could make four different types of response when a target was presented (or when the target was absent); hits, misses, false alarms, and correct rejections (see Table 8.1). For example, when an auditory stimulus was presented, a trial in which a participant pressed the auditory response key would be counted as a ‘hit’, whereas a trial in which a participant failed to press the auditory response key would be counted as a ‘miss’. On trials in which no auditory stimulus was presented, a trial in which a participant 203

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

pressed the auditory response key would be counted as a ‘false alarm’, whereas a trial in which a participant refrained from pressing the auditory response key would be counted as a ‘correct rejection’. The d’ and c values were then calculated by analysing a participant’s hit and false alarm rates. d’ was calculated as the transformed z-score (under the assumption of a cumulative normal distribution) of the proportion of hits minus the transformed z-score of the proportion of false alarms; c was calculated as 0.5 multiplied by the transformed z-score of the proportion of hits plus the transformed z-score of the proportion of false alarms. For the example shown in Table 8.1, the calculations would be: d’ = z-score(0.98) - z-score(0.10) = 3.34 c = 0.5 × (z-score(0.98) + z-score(0.10)) = 0.39

Table 8.1. Table showing the four possible types of responses; hits, misses, false alarms and correct rejections. Participant’s responses Yes No Target present Hit e.g., 98% False alarm e.g., 10% Miss e.g., 2% Correct rejection e.g., 90%

Target absent

For the Colavita signal detection task, the variables of interest were the sensitivity to, and response criterion for, the auditory stimulus when it was presented by itself versus when it was presented in the presence of a visual stimulus (i.e., the d’ and c values for unimodal auditory and bimodal auditory stimuli), and the sensitivity to, and response criterion for, the visual stimulus when it was presented by itself versus when it was presented in the presence of an auditory stimulus (i.e., the d’ and c values for unimodal visual and bimodal visual stimuli). The way in which the hit rates

204

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

and false alarm rates were used to calculate the d’ and c values for the variables of interest are summarised in Table 8.2.
Table 8.2. Mean hit and false alarm rates for the unimodal auditory, unimodal visual, bimodal, and target absent trials in the Colavita task in Experiment 8.1. The participants’ sensitivity (d’) and criterion (c) values for the unimodal auditory, bimodal auditory (i.e., the auditory stimulus in the presence of a visual stimulus), unimodal visual, and bimodal visual stimuli (i.e., the visual stimulus in the presence of an auditory stimulus) were calculated using the following two calculations: [d’ = z-score(proportion hits) - z-score(proportion false alarms)] and [c = 0.5 × (z-score(proportion hits) + z-score(proportion false alarms))]. Thus, a participant’s sensitivity to unimodal auditory stimuli was calculated by subtracting the z-score of the proportion of auditory false alarms made on target absent trials from the z-score of the proportion of auditory hits on unimodal auditory trials. Hit and false alarm rates Hits False alarms

Stimulus

Signal detection measures

Unimodal auditory Comparison stimuli % hits and false alarms d’ c Bimodal auditory Comparison stimuli % hits and false alarms d’ c Unimodal visual Comparison stimuli % hits and false alarms d’ c Bimodal visual Comparison stimuli % hits and false alarms d’ c

Unimodal auditory (auditory hits) 62.4 (5.2)

Target Absent (auditory false alarms) 2.5 (0.6) 2.5 (0.2) 0.8 (0.1)

Bimodal (auditory hits) 65.0 (5.0)

Unimodal visual (auditory false alarms) 7.5 (2.6) 2.2 (0.2) 0.6 (0.1)

Unimodal visual (visual hits) 64.1 (5.7)

Target Absent (visual false alarms) 1.0 (0.2) 2.8 (0.2) 0.9 (0.1)

Bimodal (visual hits) 75.2 (4.4)

Unimodal auditory (visual false alarms) 3.9 (1.0) 2.8 (0.2) 0.5 (0.1)

It should be noted that the average hit and false alarm values shown in the second and third columns in Table 8.2 do not appear to match with the d’ and c values given in the fourth column; for example, the sensitivity (d’) to unimodal auditory stimuli calculated from the average hit and false alarm rates across the participants [zscore (0.624)-z-score(0.025) = 2.26] is not equal to the d’ measure given in Table 8.2

205

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

(d’ = 2.5). The reason for this discrepancy is because the d’ values shown in Table 8.2 represent the average of the d’ scores which were calculated on a participant-byparticipant basis (remember that the z-transformation is non-linear; Table 8.3 illustrates this point).
Table 8.3. The proportion of hits and false alarms, the d’ score (calculated on a participant-by-participant basis), and the average value for these measures, for four sample participants. The data in Table 8.3 demonstrates how the d’ score calculated from the average hit and average false alarm rates across participants can be different from the average of the d’ scores calculated on a participant-byparticipant basis. For this example, the d’ calculated from the average scores (a hit rate of 0.974 and a false alarm rate of 0.074; see the values in the table) is 3.48, which is different from the average of the d’ values which were calculated on a participant-by-participant basis (i.e., 3.87). Proportion hits Participant 1 2 3 4 Average scores Proportion false alarms d’

0.960 0.993 0.950 0.993 0.974

0.200 0.080 0.007 0.007 0.074

2.59 3.86 4.10 4.91 3.87

Signal detection analyses. The sensitivity (d’) data from the Colavita task (see Table 8.2 and Figure 8.1) were analysed using an ANOVA with the factors of Target Modality (Auditory or Visual), and Target Type (Unimodal or Bimodal). This analysis revealed a significant main effect of Target Type [F(1, 21) = 8.89, p = .007], attributable to d-prime being higher on unimodal than on bimodal trials (d’ = 2.65 vs. 2.48, respectively). The numerical trend toward lower d-prime scores for auditory than for visual stimuli (d’ = 2.34 vs. 2.80, respectively), was only marginally significant [F(1, 21) = 4.11, p = .065], and can furthermore be explained by the interaction between Target Modality and Target Type [F(1, 21) = 5.80, p = .025]. In the bimodal condition, d’ was significantly lower for auditory than for visual targets (see Figure 8.1; t(21) = 2.92, p = .008), whereas this difference failed to reach 206

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

statistical significance in the unimodal condition (t(21) = 1.11, p = .279). That is, on the bimodal trials (but not on the unimodal trials), participants were significantly less sensitive to auditory than to visual stimuli. Significantly lower d’ values were also observed for the auditory targets in the bimodal condition than auditory targets in the unimodal condition (t(21) = 4.34, p < .001), whereas no such effect was observed for visual targets (t(21) = 0.04, p = .972).
More sensitive

3.5

* ***

3

2.5

Sensitivity (d' )

2

Unimodal Bimodal

1.5

1

0.5

0
Less sensitive

* ***
Auditory targets Visual targets

p < .05 p < .001

Figure 8.1. The mean sensitivity (d’) values for the unimodal auditory, unimodal visual, bimodal auditory, and bimodal visual targets in the Colavita task. The error bars indicate the standard errors of the means.

A similar ANOVA performed on the criterion (c) data revealed a significant main effect of Target Type [F(1, 21) = 40.38, p < .001], with lower values of c (indicating that participants were less conservative in their responding) observed on the bimodal than on the unimodal target trials (c = 0.56 vs. 0.87, respectively). The significant interaction between Target Modality and Target Type [F(1, 21) = 8.63, p = .008], was caused by the value of c being significantly lower in the bimodal than in the unimodal conditions for both the visual targets (see Figure 8.2; t(21) = 9.31, p < .001), and the auditory targets (t(21) = 2.69, p = .015); this effect was larger for the visual targets (t(21) = 2.94, p = .008). There was no difference in the criterion for 207

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

responses to the unimodal auditory and visual targets (t(21) = 1.24, p = .226), or between bimodal auditory and visual targets (t(21) = 0.63, p = .534). No main effect of Target Modality [F < 1, n.s.] was obtained.

More conservative

1.2

* ***

1

0.8

Criterion (c )

0.6

Auditory targets Visual targets

0.4

0.2

0
More liberal

Unimodal

Bimodal

* ***

p < .05 p < .001

Figure 8.2. The mean criterion (c) values for the unimodal auditory, unimodal visual, bimodal auditory, and bimodal visual targets in the Colavita task. The error bars indicate the standard errors of the means.

RT data. A similar analysis of the RT data was performed on those trials where the participants responded correctly. Although the Colavita task was unspeeded, the RT data can be taken to provide an additional measure of the difficulty of the task under conditions of unimodal and bimodal stimulus presentation. The analysis revealed a significant main effect of Target Type [F(1, 21) = 5.15, p = .034], attributable to participants responding more rapidly on the unimodal trials (748ms) than on bimodal target trials (770ms). There was also an interaction between Target Modality and Target Type [F(1, 21) = 5.48, p = .029], attributable to participants responding less rapidly to auditory targets presented bimodally than to those presented unimodally (770ms vs. 730ms; t(21) = 2.84, p = .010), whereas there was 208

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

no significant differences in RTs to visual targets presented bimodally versus unimodally (770ms vs. 766ms; t(21) = .37, p = .714). There was no main effect of Target Modality [F(1, 21) = 1.41, p = .249].

8.1.3 Discussion The results of Experiment 8.1 highlight a clear Colavita effect; that is, when the participants failed to respond correctly on the bimodal target trials (36.6% of all bimodal trials), they made significantly more visual-only than auditory-only responses (25.4% vs. 11.2% of all bimodal trials). The percentage of errors made on bimodal trials was much larger in Experiment 8.1 (36.6% errors) than in Experiment 2.1 (16.5% errors; t(32.97) = 4.45, p < .001). This is probably due to the fact that in the SDT Colavita task, the stimuli were presented at the 75% threshold in this study, and so the participants were more likely to fail to detect the stimuli (than when the stimuli were presented well above threshold). Analysis of the d’ data revealed that participants were not significantly less sensitive to auditory stimuli than to visual stimuli when they were presented unimodally, but were significantly less sensitive to auditory than to visual stimuli when they were presented bimodally (where there was no significant difference in participants’ sensitivity to unimodal than bimodal visual stimuli). In other words, participants’ sensitivity to auditory stimuli decreased in the presence of a concurrent visual stimulus. This result supports the hypothesis that there is a genuine perceptual contribution to the Colavita effect, resulting from the lower sensitivity to auditory stimuli that was observed in the presence of the simultaneous visual stimuli on the bimodal target trials. The fact that participants’ RTs to the auditory stimuli were also

209

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

slower in the presence of a concurrent visual stimulus is also consistent with this observation. Turning now to the criterion data, one way in which a criterion shift could have contributed to the Colavita effect would have been if a decrease in sensitivity to the auditory stimulus were to have led to participants adopting a higher (i.e., more conservative) response criterion for the auditory stimulus (cf. Gorea & Sagi, 2002). This might also have been expected to lead to fewer auditory responses on the bimodal trials. Since the participants were less sensitive to bimodal auditory than to bimodal visual targets, it was thought that participants might therefore adopt a more conservative criterion when responding to bimodal auditory stimuli than when responding to bimodal visual stimuli. However, the results revealed no significant differences in criterion between the auditory and visual bimodal stimuli. (Instead, participants responded more liberally on the bimodal trials than on the unimodal target trials, where this effect was larger for visual than for auditory targets.) These results therefore suggest that the Colavita effect observed in Experiment 8.1 was not caused by participants changing their criterion for responding to the auditory stimulus when presented concurrently with a visual stimulus (or vice versa).

8.2

GENERAL DISCUSSION
The experiment reported in this chapter represents the first time that a signal

detection analysis has ever been performed on the data from a Colavita task. The primary aim of Experiment 8.1 was to investigate the contributions of perceptual (sensitivity shifts) and decisional (shifts in response criteria) factors to the Colavita effect. The results of the Colavita task revealed a significant Colavita effect. The

210

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

results of the SDT analysis revealed that while participants exhibited a significant decrease in sensitivity to auditory stimuli when they were presented concurrently with visual stimuli, participants did not respond more conservatively (i.e., their criterion was not significantly larger) for bimodal auditory stimuli than for bimodal visual stimuli, suggesting that the Colavita visual dominance effect observed in this study reflects a perceptual (rather than a decisional) phenomenon. The results of Experiment 8.1 suggest that decisional (e.g., criterion shifting effects) factors do not contribute to the Colavita effect; at least for the Colavita effect that is observed when the experimental confounds (e.g., response biases etc, that were present in the early research) have been eliminated. This can provide one possible explanation for why the magnitude of the Colavita effect found in the experiments reported in this thesis is much smaller than that reported in Colavita’s original study (Colavita, 1974; see Chapter 1, General discussion, for a discussion of this comparison). It is possible that decisional factors may have contributed to, and therefore increased, the magnitude of the Colavita effect reported in the early studies of the phenomenon (e.g., Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979; Egeth & Sager, 1977), whereas only a small Colavita effect (which is most likely due to perceptual effects) remains when the experimental confounds have been eliminated. The results of Experiment 8.1 also suggested that the Colavita effect is contributed to by a decrease in sensitivity toward the auditory stimulus (when it was presented concurrently with a visual stimulus). The question that arises here is whether the decrease in sensitivity toward the auditory stimulus (when it was presented in the presence of a visual stimulus) was due to a decrease in the strength of the auditory signal, an increase in the amount of internal noise, both, or even an

211

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

increase in the strength of the auditory signal and a relatively larger increase in the amount of internal noise (remember, sensitivity gives an indication of how easily a participant can differentiate the target signal from the internal noise). Work by Odgaard, Arieh, and Marks (2004) may shed light on this question. Odgaard et al. (2004) demonstrated that participants tend to rate a simple auditory stimulus (a 40ms white noise burst) presented with a simultaneous light stimulus as being more intense than when the auditory stimulus is presented alone (Odgaard et al. argue that this effect was not due to response biases, but rather due to an early sensory process). Thus, Odgaard et al.’s results suggest that the strength of the auditory stimulus signal increases in the presence of a visual stimulus. It is possible, however, that the presence of the visual stimulus may also increase the amount of internal noise associated with the auditory signal (note that Odgaard et al. did not measure whether the internal noise associated with the auditory signal had increased; i.e., they did not present trials in which no auditory stimulus was presented and measure the auditory false alarm rates). Indeed, the finding that visual stimuli can activate auditory processing areas even when no auditory stimulus is presented (e.g., in silent speech; Bernstein, Auer, Moore, Ponton, Don, & Singh, 2002; Olson, Gatenby, & Gore, 2002; Paulesu, Perani, Blasi, Silani, Borghese, De Giovanni, Sensolo, & Fazio, 2003) suggests that it is possible for a visual stimulus to increase the internal noise associated with an auditory stimulus (i.e., the presentation of a visual stimulus could increase the likelihood that participants will make an auditory response even when no auditory stimulus is present). In addition, the finding that across a number of experiments reported in this thesis (Experiments 2.2, 3.1, 3.2, 7.1, and 7.2) participants made more unimodal auditory errors than unimodal visual errors (i.e., they pressed the auditory response key when a visual stimulus was presented

212

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

alone more often than they pressed the visual response key when an auditory stimulus was presented alone) is, again, consistent with this notion that the visual stimuli may have been increasing the amount of internal noise associated with the auditory signal.
Figure 8.3. A schematic diagram of the internal response and its probability of occurrence for the auditory signal and its associated noise component, for unimodal auditory stimuli (A), and bimodal auditory stimuli (B). If, during the presentation of a bimodal stimulus (B), both the auditory signal and noise increase (with the noise increasing relatively more than the auditory signal), and if the criterion is set at the level of the internal responses, then both the d-prime value and the criterion would decrease (see B).

A
Signal
criterion

Probability

Noise

B

Internal response

Probability

criterion

Internal response

Thus, if, in Experiment 8.1, a concurrently-presented visual stimulus increased both the auditory signal strength and the amount of internal noise associated with the auditory signal (where the increase in internal noise was larger than the increase in signal strength), then one would expect both the d-prime value and the criterion to decrease during bimodal stimulation (for example, if participants had set their criterion for responding to the auditory stimulus according to the auditory signal strength; see Figure 8.3). Indeed, the participants’ d-prime and criterion values were lower for bimodal auditory than for unimodal auditory stimuli, thus supporting the notion that the decrease in participants’ sensitivity to auditory stimuli during bimodal

213

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

stimulation was caused by both an increase in the auditory signal strength as well as a relatively larger increase in the amount of internal noise.

Figure 8.4. A schematic diagram of the internal response and its probability of occurrence for the visual signal and its associated noise component, for unimodal visual stimuli (A), and bimodal visual stimuli (B).

A
Signal

Probability

criterion

Noise

B

Internal response

Probability

criterion

Internal response

Another question to arise here is why there was no change in sensitivity to the visual stimulus when it was presented together with an auditory stimulus. Stein, London, Wilkinson, and Price (1996) claimed to have demonstrated that a concurrent sound can increase a participant’s perception of the intensity of a visual signal. However, Odgaard, Arieh, and Marks (2003) argued that Stein et al.’s results reflected a decisional effect, and that a concurrent sound would not increase a participant’s perception of the intensity of a visual signal when the influence of response biases were removed. Thus, Stein et al.’s and Odgaard et al.’s results could be taken to suggest that a concurrently-presented auditory stimulus may not change the visual signal strength or amount of internal noise (i.e., a concurrently-presented auditory stimulus would not affect a participant’s sensitivity to a visual stimulus), but may

214

CHAPTER 8 - A SIGNAL DETECTION STUDY OF THE COLAVITA EFFECT

cause participants’ criterion for responding to a visual stimulus to decrease (see Figure 8.4 for an illustration of this). Indeed, participants in Experiment 8.1 showed no change in d-prime, but a decrease in criterion, for visual stimuli during bimodal stimulation. In summary, the results reported in this chapter provide the first empirical evidence to demonstrate that the Colavita effect may be caused, at least in part, by a decrease in sensitivity to the auditory stimulus when it is presented concurrently with a visual stimulus (at least on trials in which the auditory and visual stimuli are presented at their 75% detection threshold). This decrease in sensitivity might be caused by the presentation of the visual stimulus increasing the internal noise associated with the auditory stimulus relative to its signal strength. At this point, however, one caveat to note is that in all of the other experiments on the Colavita effect (apart from Experiment 8.1), the auditory and visual stimuli were presented well above threshold levels. Given that interactions occurring between stimuli presented above threshold may not be so affected by the relative perceived intensities of the stimuli (Spence et al., 2001b), it may therefore be the case that any decrease in sensitivity to the auditory stimulus on bimodal trials in which the stimuli are presented well above threshold may not contribute significantly to the Colavita effect.

215

CHAPTER 9 - DISCUSSION

CHAPTER 9
9.1 KEY FINDINGS
The Colavita effect represents an example of visual dominance which provides a remarkable demonstration of how a visual stimulus can dominate a person’s perception to such an extent that it can, on occasion, preclude their perception of a concurrently-presented auditory stimulus. However, despite having been discovered more than three decades ago, the Colavita visual dominance effect has remained something of a conundrum to cognitive science researchers. The primary aim of the research reported in this thesis was therefore to investigate some of the key factors modulating the Colavita visual dominance effect in order to try and gain a better understanding of the perceptual mechanisms behind this fascinating phenomenon. The hope being that the insights gained by exploring the causes of the Colavita effect might provide a useful contribution to the wider area of research on visual dominance and extinction. The main findings to emerge from the seven experimental chapters will be summarized first, after which an explanation describing how the Colavita effect occurs will be put forward. There are a number of problematic factors that have made a straightforward interpretation of the results of the earliest studies of the Colavita effect difficult (Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979; Egeth & Sager, 216

CHAPTER 9 - DISCUSSION

1977; Johnson & Shapiro, 1989; Shapiro et al., 1984; see Chapter 1, Section 1.4, for a discussion). These factors include deceit by the experimenter regarding the types of stimuli that participants should expect to be presented, the presentation of auditory and visual stimuli from different spatial locations, difficulties in interpreting participants’ responses, and the relatively infrequent presentation of bimodal targets. The purpose of the experiments reported in Chapters 2 and 3 of this thesis was therefore to find the optimal methodology (in terms of finding an appropriate set of response requirements for responding to the three types of target) with which to investigate the Colavita effect, and also to ascertain whether the effect reflected a genuine phenomenon (i.e., a phenomenon that was not caused by response biases or by any of the other aforementioned factors). The participants were therefore presented with a version of the Colavita task which was free from those confounds, and in which the response requirements (the two-key or three-key response conditions1; Chapter 2) or the relative frequency with which the bimodal targets were presented (10%, 33%, 50%, 60%, or 90% of the trials; Chapter 3) were manipulated. The main finding to emerge from the experiments reported in Chapter 2 was that the Colavita effect can be demonstrated under both two-key and three-key response conditions. The fact that the Colavita effect emerged regardless of the particular response requirements of the task (especially given that the use of the two-

In both the two-key and three-key response conditions, the participants were instructed to press the auditory response key in response to a unimodal auditory stimulus, and the visual response key in response to a unimodal visual stimulus. In response to a bimodal stimulus, participants in the two-key response condition were clearly instructed to press both the auditory and visual response keys, whereas in the three-key response condition the participants were instructed to press a third, separate bimodal response key instead. These response requirements contrast with the previously used response requirements where, for example, participants were instructed to respond to the signal they perceived first (e.g., Colavita, 1974; Colavita et al., 1976; Egeth & Sager, 1977; Johnson & Shapiro, 1989; Quinlan, 2000; Shapiro et al., 1984). The advantage of the two- and three-key response requirement versions of the task is that they did not necessitate participants having to explicitly decide which one of the two components to respond to (which could have introduced response biases into the experimental design).

1

217

CHAPTER 9 - DISCUSSION

key and three-key response requirements decreased the potential contribution of response biases to the emergence of the Colavita effect) suggests that the Colavita effect is not simply an artefact of the response requirements of the particular task given to participants. In addition, the results of the experiments reported in Chapter 2 also ruled out the possibility that the Colavita effect was due to response biases or due simply to participants’ response latencies being slower for auditory than for visual stimuli 2. The main finding to emerge from the experiments reported in Chapter 3 was that the Colavita effect emerged no matter whether the bimodal targets constituted 33%, 50%, or 60% of the trials in a given block of trials. This result therefore suggests that the Colavita effect (and the apparent difficulty that participants seem to have in responding to both components of a bimodal target) is not simply attributable to the lower frequency with which the bimodal targets have typically been presented in previous research. Instead, the Colavita effect appears to emerge because of a difficulty that participants have in the perceptual processing of, or response selection toward, the auditory component of the bimodal stimulus. In summary, the experimental findings reported in Chapters 2 and 3 suggest that the Colavita effect represents a perceptually-based phenomenon (one that does not simply reflect the consequences of response biases, the response requirements of the task, etc). Having established this, the aim of the experiments reported in the subsequent chapters was to uncover some of the factors modulating the Colavita effect.

Given that the RTs to visual stimuli were often faster than the RTs to auditory stimuli (note that this was a serendipitous, rather than planned, outcome of the results), it could be argued that it is simply the case that the stimulus that participants respond to fastest dominates their responding. However, the Colavita effect has even been reported under conditions where the auditory RTs were faster than the visual RTs (Sinnett et al., 2007; see also Experiment 2.2), or when there was no significant difference between the RTs to auditory and visual stimuli (Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979). Hence, the Colavita effect cannot simply be attributed to participants responding more rapidly to the visual component of bimodal target stimuli.

2

218

CHAPTER 9 - DISCUSSION

In Chapter 4, the contribution of attention to the Colavita effect was examined. Those researchers who conducted some of the early studies of the Colavita effect postulated that the failure by participants to respond to the auditory component of the bimodal targets in the Colavita task may have been caused by participants having a tendency to endogenously attend to the visual modality (Colavita & Weisberg, 1979; Egeth & Sager, 1977; Posner et al., 1976). Another explanation of the Colavita effect, one that was originally proposed in Chapter 4, is that it is caused by the greater capacity of a visual stimulus, than an auditory stimulus, to capture attention exogenously (i.e., a visual stimulus can attract attention exogenously away from audition and toward the visual modality; see Hamlin, 1895; Rodway, 2005; Smith, 1933; Spence et al., 2001a; Turatto et al., 2002). According to this explanation, the Colavita effect could be caused by participants’ attention being exogenously captured by the visual component of the bimodal stimulus. The aim of the three experiments reported in Chapter 4 was therefore to investigate the influence of attention (directed endogenously, exogenously, or both endogenously and exogenously, toward auditory and/or vision) on the Colavita effect. The focus of participants’ attention was endogenously manipulated by instructing them to expect auditory or visual stimuli throughout a whole block of trials, while their attention was exogenously manipulated by presenting them with an auditory or a visual exogenous cue shortly before the onset of the target stimulus. The principal result to emerge from the three experiments reported in Chapter 4 was that the magnitude of the Colavita effect was significantly affected by both the attentional manipulations (i.e., whether participants’ attention was manipulated endogenously or exogenously). In particular, the results of Chapter 4 supported the argument that people tend to attend endogenously to vision, and that this tendency contributes to the

219

CHAPTER 9 - DISCUSSION

emergence of the Colavita effect. They also demonstrated that visual stimuli may have a greater capacity to exogenously attract attention than auditory stimuli (also see Turatto et al., 2002), and suggest that this is also a factor that contributes to the emergence of the Colavita effect. One of the explanations put forward in Chapter 4 for how attention might modulate the Colavita effect was in terms of the law of prior entry (see James, 1890; Mollon & Perkins, 1996; Spence et al., 2001b; Titchener, 1908). The tendency of participants to have their attention directed toward the visual modality (either endogenously and/or exogenously) would, according to the law of prior entry, result in them perceiving the visual component of the bimodal target prior to the auditory component3. Thus, the delayed perception of the auditory stimulus may have been responsible for participants’ failure to respond to it. One of the major aims of the experiment reported in Chapter 5 was therefore to investigate whether the Colavita effect would be affected by which stimulus participants perceived first. If a delayed perception of the auditory stimulus contributes to the Colavita effect, then the

Note, however, that the directing of participants’ endogenous/exogenous attention toward vision does not by itself necessitate participants perceiving the visual stimulus first; the prior entry of the visual stimulus depends on participants’ PSS prior to the attentional manipulation, as well as the effectiveness of the manipulation itself. Given that participants’ PSS was calculated to be approximately +12ms (see Chapter 5, Section 5.1.2; the visual stimulus had to lead by 12ms in order for participants to perceive the auditory and visual stimuli as being presented simultaneously), and that attentional manipulations can shift participants’ PSS values in the order of 30-100ms (e.g., Spence et al., 2001b), it seems likely that participants directing their attention toward vision would result in a sufficient shift of participants’ PSSs to elicit the prior entry of the visual stimulus in the Colavita paradigms used throughout this thesis. Furthermore, the fact that participants responded faster (on average 40ms faster) to targets presented in the stimulus modality toward which their attention was directed, where RTs provide a rough indication of the temporal order in which the participants perceive the stimuli (see Miller & Schwarz, 2006), again, supports this notion that directing a participants’ attention toward a particular modality resulted in the prior entry of a stimulus presented in that modality. In sum, although the experiments reported in this thesis do not provide any direct evidence for the claim that there was prior entry of the visual stimulus on the bimodal trials in which the auditory and visual stimuli were presented simultaneously, the fact that participants responded faster to the visual than to the auditory component of the bimodal stimulus and that they appeared to attend endogenously/exogenously toward the visual modality (where this would result in prior entry of the visual stimulus) provides support for this claim. It is therefore plausible to suggest that participants’ tendency to have their attention directed toward the visual modality would result in participants having a delayed perception of the auditory stimulus.

3

220

CHAPTER 9 - DISCUSSION

magnitude of the Colavita effect should increase under conditions where the visual stimulus was presented first. Experiment 5.1 explored whether the magnitude of the Colavita effect would be modulated by the temporal order in which the auditory and visual stimuli were presented and by the magnitude of the temporal separation between the two stimuli. This was achieved by presenting the auditory and visual components of the bimodal stimuli at one of a range of SOAs. The secondary aim of Experiment 5.1 was to determine whether the Colavita effect could be attributed to participants having a generally delayed perception of the auditory stimulus relative to that of the visual stimulus on the bimodal trials (which was investigated by means of a crossmodal TOJ task). The results revealed, firstly (and somewhat surprisingly), that the participants did not have a generally delayed perception of the auditory stimulus4 (at least when they were performing a crossmodal TOJ task), and, secondly, that the Colavita effect was modulated by the temporal order in which the auditory and visual stimuli were presented on the bimodal target trials. Specifically, the Colavita effect was larger when the visual stimulus was presented first, and reversed/attenuated when the auditory stimulus led, thus supporting the notion that a delayed perception of the auditory stimulus could contribute to the emergence of the Colavita effect (as first suggested in Chapter 4). The third finding to emerge from Experiment 5.1 was that the Colavita effect was modulated by the temporal separation between the auditory and visual stimuli. In particular, the Colavita effect was eliminated when the temporal
This result was particularly surprising given that it has been shown that patients suffering from extinction (a phenomenon that shares a number of similarities with the Colavita effect; see Section 9.3, for a detailed account of the similarities and differences between these two phenomena) exhibit a delayed perception of the extinguished stimulus (Baylis et al., 2002; Guerrini et al., 2003; Rorden et al., 1997; Sinnett et al., 2007). A number of researchers have argued that the delayed processing of the extinguished stimulus may itself contribute to the phenomenon of extinction (Bueti et al., 2004; di Pellegrino et al., 1997a). It was therefore predicted that participants in the Colavita task may similarly have had a generally delayed perception of the (extinguished) auditory stimulus.
4

221

CHAPTER 9 - DISCUSSION

separation between the auditory and the visual stimuli was large enough so that participants could respond to them as if they were two sequentially presented unimodal stimuli. The magnitude of the Colavita effect did not, however, appear to be modulated by the extent to which participants temporally bound the auditory and visual stimuli, thus failing to provide any support for the notion that the unity effect contributes to the Colavita effect. One of the main aims of the experiments reported in Chapters 5, 6, and 7 was to investigate whether those factors contributing to the assumption of unity between the auditory and visual stimuli on the bimodal trials (i.e., the extent to which participants perceived them as belonging to a single unitary multisensory event) would modulate the magnitude of the Colavita effect. It has been argued elsewhere that the stronger the assumption of unity that a participant has about two unisensory events, the greater the intersensory bias/binding between them (Bedford, 2001; Welch & Warren, 1980; see Spence, 2007, for a recent review). Thus, it was predicted that the Colavita effect would be modulated by the extent to which participants perceived the auditory and visual stimuli as constituting a single audiovisual event versus perceiving them as two distinct perceptual events. The assumption of unity between unimodal auditory and visual stimuli has been shown to depend on their spatial and temporal separation (see Calvert et al., 2004; Vatakis & Spence, 2007a, for reviews), and has also sometimes been shown to be modulated by their semantic congruency as well (see Laurienti et al., 2004; Molholm et al., 2004; Taylor et al., 2006; Vatakis & Spence, 2007a). The influence of the temporal and spatial separation between the stimuli, and their semantic congruency on the magnitude of the Colavita effect was therefore investigated in the experiments reported in Chapters 5 through 7 (in which

222

CHAPTER 9 - DISCUSSION

the temporal separation, spatial separation, and semantic congruency between the auditory and visual stimuli were varied). The principal finding to emerge from the experiments reported in these chapters was that the temporal and spatial separation between the auditory and visual components of a bimodal stimulus can modulate the Colavita effect, while the semantic congruency between the auditory and visual stimuli does not modulate the magnitude of the Colavita effect. The effect of temporal separation on the magnitude of the Colavita effect was explained in terms of the Colavita effect being attenuated (or eliminated) when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli. Meanwhile, the effect of spatial separation on the magnitude of the Colavita effect was explained in terms of the Colavita effect being attenuated (or eliminated) when participants have a redundant spatial cue informing them that two stimuli had been presented. The effects of temporal and spatial separation on the magnitude of the Colavita effect were not, however, explained in terms of the Colavita effect being modulated by the extent to which participants temporally or spatially bound the auditory and visual stimuli. Thus, the results of Chapters 5-7 do not necessarily support the notion that the assumption of unity between the auditory and visual stimuli on bimodal trials modulates the Colavita effect. Finally, while the majority of the chapters in the thesis focused on determining the factors that modulate the Colavita visual dominance effect, the aim of the experiment reported in Chapter 8 was to explore the processes behind the effect. Signal detection theory was used to quantitatively evaluate the differential contributions of perceptual (sensitivity shifts) and decisional (shifts in response criteria) factors to the Colavita visual dominance effect. Participants were presented

223

CHAPTER 9 - DISCUSSION

with a version of the Colavita task that had been modified to allow for SDT analyses (where the auditory and visual stimuli were presented at each participant’s individually-determined 75% detection threshold). The main finding to emerge from Experiment 8.1 was that there was a significant decrease in participants’ sensitivity (but no change in their response criterion) to auditory stimuli when they were presented concurrently with visual stimuli, thus suggesting that the Colavita visual dominance effect (at least for the effects observed in Experiment 8.1) reflected a perceptual (rather than a decisional) phenomenon. It was suggested that this decrease in sensitivity might be caused by the presentation of the visual stimulus increasing the internal noise associated with the auditory stimulus relative to its signal strength. It was, however, noted that the auditory and visual stimuli were presented well above threshold in all of the other experiments reported in this thesis, which may mean that any decrease in sensitivity to the auditory stimulus on those bimodal trials may not have contributed significantly to the Colavita effects reported there. In sum, the results that emerged from the experiments reported in Chapters 2 and 3 suggested that the Colavita effect is a perceptually-based phenomenon (that is, it is not caused by response biases, the response requirements of the task, etc). The results of the experiments reported in Chapter 4 revealed that directing participants’ attention toward the visual modality can increase the Colavita effect. The results reported in Chapter 5 revealed that the Colavita effect is larger when the visual stimulus is presented first (on bimodal trials in which the auditory and visual stimuli are presented asynchronously). The experiments reported in Chapters 5, 6, and 7 provide no clear support for the notion that the unity effect can modulate the Colavita effect. Instead, the experiment reported in Chapter 5 suggests that the Colavita effect is attenuated or eliminated when participants can respond to the components of the

224

CHAPTER 9 - DISCUSSION

bimodal stimulus as if they were sequentially presented unimodal stimul, while the experiment reported in Chapter 6 suggests that Colavita effect is attenuated or eliminated when participants have a redundant spatial cue informing them that two stimuli had been presented. Finally, the results of the Experiment 8.1, performed to determine the differential contributions of perceptual and decisional factors to the Colavita effect, suggested that the Colavita effect reflects a perceptual (rather than a decisional) phenomenon.

9.2

AN EXPLANATION OF THE COLAVITA EFFECT
Three main findings emerged from the experiments reported in this thesis.

First, participants’ tendency to have their attention directed toward the visual modality contributes to the Colavita effect. Second, the Colavita effect is larger when the visual component of the bimodal stimulus is presented prior to the auditory component (than vice versa). Third, the Colavita effect is attenuated or eliminated when participants can respond to the components of the bimodal stimulus as if they were sequentially presented unimodal stimuli, or when participants have a redundant spatial cue informing them that two stimuli had been presented. Having ascertained a number of the main factors that contribute to the Colavita effect, a description is now put forward to explain how the visual component of the bimodal stimulus may sometimes preclude the perception of the auditory component, and what happens on those trials in which participants do respond to the auditory stimulus (see Figure 9.1 for a schematic representation of this explanation).

225

CHAPTER 9 - DISCUSSION

Bimodal stimulus: the presentation of auditory and visual stimuli

A

Attention directed endogenously or exogenously toward audition

Attention directed endogenously or exogenously toward vision

B

Participants respond sequentially to the auditory and visual stimuli Participants use a redundant spatial cue to infer that two stimuli have been presented

C

Prior entry of the visual stimulus The auditory stimulus arrives during the attentional dwell time of the visual stimulus

D

The auditory stimulus is replayed in echoic memory Participants delay their auditory or visual responses

The auditory stimulus information decays

Figure 9.1. Schematic representation summarizing how the visual component of the bimodal stimulus may preclude the perception of the auditory component, and what happens on those trials in which participants respond appropriately to the auditory stimulus. A) Participants will respond to the auditory stimulus when their attention is endogenously or exogenous directed toward audition. B) Participants will respond accurately to both components of the bimodal stimulus if they have the opportunity to respond to the stimuli sequentially (for example, if there is large temporal separation between them). If the stimuli are presented from different spatial positions, participants can also use the redundant spatial cue informing then that two stimuli had been presented to infer that a bimodal stimulus must have been presented. C) If participants have their attention directed either endogenously or exogenously toward the visual modality, they will have a delayed perception of the auditory stimulus. Hence, the auditory stimulus will arrive during the attentional dwell time of the visual stimulus. D) If participants have a delayed perception of the auditory stimulus, if they fail to consolidate the auditory stimulus information in time (for example, due to their limited-capacity attentional resources being allocated to the processing of the visual stimulus which they perceived first), the representation of the auditory stimulus would consequently decay and may result in participants’ failure to report it. However, participants are able to respond to the auditory component of the bimodal stimulus if they delay either their auditory or their visual response (in order to make a response couplet) or if they postpone the processing of the auditory stimulus (until after they have responded to the visual stimulus) by storing it in their echoic memory.

226

CHAPTER 9 - DISCUSSION

9.2.1 An explanation for how the visual stimulus dominates 9.2.1.1 The role of endogenous and exogenous attention Over the years, many researchers have argued that people tend to direct their attention endogenously toward the visual modality (e.g., Posner et al., 1976; Spence et al., 2001a). It has been suggested that the Colavita effect arises from the tendency of participants to endogenously attend toward vision (Colavita & Weisberg, 1979; Egeth & Sager, 1977). Another factor that may contribute to the Colavita effect is the greater capacity of visual (than auditory) stimuli to capture a person’s attention exogenously (i.e., attention is exogenously drawn away from the auditory stimulus as a result of having been attracted toward the visual stimulus; e.g., Hamlin, 1895; Rodway, 2005; Turatto et al., 2002; Smith, 1933; Spence et al., 2001a); it was suggested in Chapter 4 that the Colavita effect might be caused by participants’ attention being exogenously captured by the visual component of the bimodal stimulus. The results of the experiments reported in Chapter 4 supported the argument that people do indeed tend to endogenously attend toward the visual modality, and the results suggest that this tendency contributes to the emergence of the Colavita effect (see Section 9.1). They also support the idea that visual stimuli have a greater capacity than auditory stimuli to attract a participant’s attention exogenously, and that this may also be a factor contributing to the emergence of the Colavita effect. Participants’ tendency to have their attention directed toward the visual modality would, according to the law of prior entry (e.g., Spence et al., 2001b; Titchener, 1908), be expected to result in participants perceiving the visual component of the bimodal target prior to the auditory component. It may therefore be possible that participants’ failure to respond to the auditory component of the bimodal stimulus might be caused, at least in part, by participants’ delayed perception of the

227

CHAPTER 9 - DISCUSSION

auditory stimulus. The fact that participants responded faster to targets presented in the stimulus modality toward which their attention was directed5, supports this notion that directing a participants’ attention toward a particular modality resulted in the prior entry of a stimulus presented in that modality6. Furthermore, the fact that the size of the Colavita effect increased when the visual stimulus was presented first (and was reduced, or even eliminated, when the auditory stimulus was presented first; see Chapter 5, Section 5.2.1) supports the notion that the Colavita effect would be increased if participants had a delayed perception of the auditory stimulus. In sum, participants’ tendency to have their attention directed, either endogenously and/or exogenously, toward the visual modality would be expected to result in participants having a delayed perception of the auditory stimulus (which has been shown to increase the magnitude of the Colavita effect). Next, the phenomenon of attentional dwell time is discussed as it may offer a putative explanation for how the delayed perception of the auditory stimulus can contribute to the Colavita effect.

9.2.1.2 Attentional dwell time and the delayed perception of the auditory stimulus Attentional blink (AB) studies, in which participants have to identify two objects presented in close temporal succession (i.e., separated by up to 500ms), have

The participants responded significantly faster to auditory than to visual stimuli on the bimodal trials when their attention was directed endogenously or exogenously toward the auditory modality (mean differences = 19ms and 12ms, respectively), and more rapidly to visual than to auditory stimuli when their attention was directed endogenously or exogenously toward the visual modality (mean differences = 98ms and 34ms, respectively; see Experiments 4.1 and 4.2). 6 One caveat to note is that although faster RTs to (for example) the visual stimulus than to the auditory stimulus may indicate that participants perceived the visual stimulus earlier than the auditory stimulus, most stimulus manipulations have smaller effects on participants’ temporal order judgments than on their RTs (see Miller & Schwarz, 2006; also see Chapter 5, Section 5.1.4). Therefore, the fact that participants may have responded 98ms faster to visual than to auditory stimuli when they were endogenously attending to vision does not necessarily indicate that participants perceived the visual stimulus 98ms before the auditory stimulus; rather, the manipulation of endogenous attention would affect participants’ perception of the temporal order of the stimuli to a lesser extent than it affected their RTs.

5

228

CHAPTER 9 - DISCUSSION

shown that the first object (T1) continues to interfere with the perception of the second object (T2; participants typically fail to report T2 on approximately 15%-60% of the trials) for a certain period of time7 (i.e., up to 500ms), known as the attentional dwell time (Broadbent & Broadbent, 1987; Duncan, Ward, & Shapiro, 1994; Moore, Egeth, Berglan, & Luck, 1996; Raymond, Shapiro, & Arnell, 1992; Ward, Duncan, & Shapiro, 1997; see Theeuwes, Godjin, & Pratt, 2004, for a review). In addition, the AB phenomenon has been reported crossmodally between audition and vision8 (Arnell & Duncan, 2002; Arnell & Jolicoeur, 1999; Arnell & Larson, 2002; Bronkhurst, van der Hoeven, Theeuwes, van der Burg, & Koelewiyn, 2006; Potter, Chun, Banks, & Muckenhoupt, 1998; for a review, see Arnell & Jenkins, 2004). That is, the presentation of a visual T1 will interfere with the perception of a subsequently presented auditory T2, and likewise, an auditory T1 will interfere with a visual T29. Thus, the crossmodal audiovisual AB is similar to the Colavita effect in terms of the fact that they both show that a visual stimulus can decrease a participants’ detection of a subsequently auditory stimulus (where it has been shown that this interference effect decreases the further apart in time the stimuli are presented, in both the AB and Colavita paradigms; see Experiment 5.1, Section 5.1.3). Hence, one possible

Note that the duration of the AB has been shown to vary between individuals (e.g., Raymond et al., 1992; Soto-Faraco & Spence, 2002). 8 Note, though, that there are mixed opinions as to whether a true AB occurs crossmodally; a number of researchers have failed to find a crossmodal attentional blink (e.g., Duncan et al., 1997; Hein, Parr, & Duncan, 2006; Potter et al., 1998; Soto-Faraco & Spence, 2002). 9 Note that while some studies have found that the VA (i.e., a visual T1 followed by an auditory T2) condition produces larger ABs than the AV modality combination (i.e., an auditory T1 followed by a visual T2), other studies have observed the opposite pattern of results (see Arnell & Jenkins, 2004, for a review). Thus, there appears to be no clear asymmetry, or dominance of the visual stimulus (in terms of the visual stimulus being resistant to ABs), in crossmodal studies of the AB phenomenon. Although this lack of visual dominance in the AB paradigm may appear to contrast with the Colavita effect in which visual dominance clearly does manifest itself, one important difference between the two paradigms is that the auditory and visual stimuli are presented simultaneously in the Colavita task (except for in Experiment 5.1), whereas the auditory and visual T1 and T2 are never presented simultaneously in the AB paradigm. Thus, any effects of visual dominance that are typically observed in the Colavita task may not be observed in the AB paradigm because the separation between T1 and T2 would be too great.

7

229

CHAPTER 9 - DISCUSSION

explanation for how the Colavita effect arises is that when participants attend toward vision (resulting in a delayed perception of the auditory stimulus relative to the visual stimulus), the auditory stimulus would arrive during the attentional dwell time of the visual stimulus and would therefore fail to be processed fully. Almost all of the available theoretical accounts of the AB (e.g., Chun & Potter, 1995; Duncan et al., 1994; Jolicoeur, Dell’Acqua, & Crebolder, 2000; Shapiro, Raymond, & Arnell, 1994) suggest that the failure to detect T2 is due to participants’ limited-capacity attentional resources temporarily being occupied with the processing of T1. In particular, according to the two-stage model of the AB (Chun & Potter, 1995), it is hypothesized that the processing of a stimulus proceeds through two stages; the first stage permits the rapid, global, and unconscious initial evaluation of the stimulus. The information from the first stage is then advanced to the second stage where limited-capacity attentional resources are required to consolidate the stimulus into amodal working memory stores, where it then becomes available for conscious report (e.g., Arnell & Jenkins, 2004; Arnell & Jolicoeur, 1999; Arnell & Larson, 2002; Jolicoeur, 1998, 1999; Jolicoeur & Dell’Acqua, 1998). There is competition between targets for prominence in the working memory stores. Furthermore, while T1 is being consolidated, T2 must wait until the limited-capacity bottleneck is freed, during which time the representation of T2 will begin to decay. Thus, if T2 is not advanced to the second stage of processing soon enough, it will decay and become unavailable for conscious report. Chun and Potter’s (1995) two-stage model has received considerable empirical support from the neurophysiological literature (e.g., Martens, Munneke, Smid, & Johnson, 2006; Sergent, Baillet, & Dehaene, 2005; Vogel & Luck, 2002; Vogel, Luck, & Shapiro, 1998; see Marois, 2005, for a review). The results of event-related potential (ERP) studies have shown that all T2 targets

230

CHAPTER 9 - DISCUSSION

(whether or not they reach consciousness) are processed to the semantic level in the brain (stage one), whereas the presence of the P3 component of the ERP, presumed to reflect the postperceptual updating of working memory (stage two), determines whether participants will respond to the T2 target or not, and hence whether there will be an AB (i.e., when participants fail to respond to T2). In sum, Chun and Potter’s (1995) two-stage model suggests that the failure to respond to T2 is due to participants’ limited-capacity attentional resources being occupied with the consolidation of T1, thus leaving the representation of T2 to decay and therefore fail to reach conscious report. Given the similarities between the audiovisual AB to the Colavita effect, the explanation put forward for the crossmodal AB may also be applicable to the visual dominance observed in the Colavita effect. The AB explanation could used to explain the Colavita effect in the following way10: If participants had their attention directed, either endogenously or exogenously, toward the visual modality (resulting in participants having a delayed perception of the auditory stimulus relative to the visual stimulus) they would then perceive the auditory stimulus as arriving during the attentional dwell time of the visual stimulus. If participants failed to consolidate the auditory stimulus information before it decays

10

One question to arise from the comparison between the audiovisual AB and the Colavita effect is why the failure to detect an auditory stimulus (T2 in the AB task, and the auditory component of the bimodal stimulus in the Colavita task) is larger in the AB paradigm as compared to in the Colavita task paradigm. For example, when a visual stimulus precedes an auditory stimulus by 100ms, participants failed to respond to the auditory stimulus on approximately 30% of the trials in the audiovisual AB tasks (see the results of Arnell & Jenkins, 2004; Arnell & Jolicoeur, 1999; Soto-Faraco & Spence, 2002), and on approximately 13% of the trials in the Colavita task (Experiment 5.1). The difference in the extent to which participants fail to detect the auditory stimulus in the audiovisual AB and Colavita tasks is most probably due to the difference between the two tasks. In the AB paradigm, T1 and T2 are presented within a rapid audiovisual stream of distractors, thus the forward or backward masking from the distractors could result in the overwriting of T2 information before it can be consolidated (Giesbrecht & Di Lollo, 1998; Maki, Couture, Frigen, & Lien, 1997). In contrast, as no masking occurs in the Colavita task (because no distractors are presented), the auditory stimulus is not prone to masking. Thus, the presence of distractors in the AB tasks may be one of the main factors that contributes to the poorer detection of auditory (T2) targets in the audiovisual AB task (as compared to the Colavita task).

231

CHAPTER 9 - DISCUSSION

(due to their limited-capacity attentional resources being temporarily occupied with the processing of the visual stimulus which they perceived first), the representation of the auditory stimulus would consequently decay, resulting in participants’ failure to report it. In order to investigate whether this explanation is applicable to the Colavita effect, it would be of great value to investigate whether the neural processes involved in the Colavita effect are similar to those observed in the crossmodal AB; one could, for example, explore whether the successful detection of the auditory component of the bimodal stimulus (in the Colavita task) would also be contingent on the elicitation of a P3 component (cf. Sergent et al., 2005). Finally, one point to note here is that there are conditions in which the AB does not occur. Namely, in conditions giving rise to the phenomenon known as Lag-1 sparing (Potter, Chun, Banks, & Muckenhoupt, 1998), the AB is absent when T2 appears immediately after T1 in the ordinal position Lag-1 (i.e., when there are no intervening targets between them). Thus, it could be argued that if participants performing the Colavita effect task have a delayed perception of the auditory stimulus relative to the visual stimulus, the auditory stimulus would effectively be appearing during the Lag-1 position and should, if the Lag-1 sparing effect occurs, therefore be perceived. However, in an extensive meta-analysis of AB studies, Visser, Bischof, and Di Lollo (1999) identified three conditions that have to be met in order for the Lag-1 sparing to be seen. First, both targets have to appear from the same spatial location; second, the temporal interval between them must not exceed the temporal window of attention; third, there should be no switch of attentional set between the two targets. As participants performing the Colavita effect would have to switch between the auditory and visual modalities (on trials in which the perception of one component of the bimodal stimulus was delayed relative to the other; note that Visser

232

CHAPTER 9 - DISCUSSION

et al., 1999, did not find any instances of Lag-1 sparing when participants had to switch sensory modalities), it would therefore be predicted that Lag-1 sparing would not occur during the performance of the Colavita effect.

9.2.2 When vision does not dominate in the Colavita task While the focus of this thesis has been mainly on the way in which the visual stimulus comes to dominate in the Colavita paradigm, it is also important to discuss what happens on the majority of trials (approximately 85% of the bimodal trials when averaged across all of the experiments reported in this thesis) in which participants successfully manage to respond to both the auditory and visual components of the bimodal stimuli.

9.2.2.1 Delaying of responses If participants had their attention directed toward the visual modality, they would have had a delayed perception of the auditory stimulus. Therefore, one way in which participants could respond would simply be to respond to the stimuli as soon as they had perceived them; that is, they may have initiated their response to the visual stimulus first before initiating their response to the auditory stimulus. Indeed, the results of the analysis of participants’ responses to the auditory and visual stimuli on bimodal trials in Experiment 2.1 demonstrates that the participants did, in fact, exhibit delayed responses to the auditory stimuli relative to the visual stimuli (on approximately 22% of the bimodal trials; see Chapter 2, Section 2.1.4). In addition, participants responded less rapidly to auditory than to visual stimuli (on the bimodal trials) in the majority of the experiments reported in this thesis. These findings, which

233

CHAPTER 9 - DISCUSSION

show that participants responded more slowly to auditory than to visual targets, support the notion that participants may have had a delayed perception of the auditory stimulus (presumably resulting from participants directing their attention toward the visual modality) on bimodal trials in the Colavita task. An alternative way in which participants could have responded to both components of a bimodal stimulus (on trials in which participants had a delayed perception of the auditory stimulus) would have been for them to delay their responses to the visual component of the bimodal stimulus (until they were ready to make the auditory response) in order to make a response couplet (Fagot & Pashler, 1992). Thus, instead of making two individual responses to the auditory and visual stimuli separately, participants would instead have made a single response to the bimodal target’s single attribute of ‘bimodality’, initiating both their auditory and visual responses at once. The fact that participants appeared to couple their responses to the auditory and visual components of the bimodal stimulus on the majority of the trials (as argued in Chapter 2, Section 2.1.4; also see Figure 2.1), as well as the finding that in the majority of the experiments reported in this thesis participants’ responses to bimodal targets were significantly slower than their responses to unimodal targets, supports the notion that participants delayed their auditory and visual responses to make response couplets on bimodal trials.

234

CHAPTER 9 - DISCUSSION

9.2.2.2 Attending to the auditory modality If, on a particular trial, participants’ attention was endogenously directed to audition11, then it would be likely that there would be prior entry of the auditory stimulus. The delayed perception of the visual stimulus would result in participants either failing to respond to the visual stimulus (because it would arrive during the attentional dwell time of the auditory stimulus), or at least making a delayed response to the visual stimulus. Based on the fact that participants tended to make more rapid responses to visual than to auditory stimuli (when their responses were not coupled), and based on the finding that participants failed to respond to the auditory stimulus more often than they failed to respond to the visual stimulus, it would therefore seem reasonable to argue that on the majority of trials participants did not endogenously direct their attention to the auditory modality.

9.2.2.3 Temporal separation and redundant spatial cues If the temporal separation between the auditory and visual stimuli was great enough so that participants would respond to the two stimuli as if they were sequentially presented unimodal stimuli, participants would be expected to respond to both stimuli and the Colavita effect would not occur. If the auditory and visual stimuli were presented from different spatial positions, participants could use the redundant spatial cue informing them that one stimulus had been presented at each of the two spatial locations in order to infer that a bimodal stimulus had been presented and thus respond to the auditory stimulus.

11

Participants might direct their endogenous attention to the auditory modality, for example, if an error on a previous trial had motivated them to direct their attention toward audition in order to avoid making another such error.

235

CHAPTER 9 - DISCUSSION

9.2.2.4 Echoic memory Finally, echoic memory (Cowan, 1984) can also contribute to participants’ ability to respond to the auditory component of a bimodal stimulus. According to Cowan (1984), echoic memory is a form of auditory memory, which can be subdivided into two types. Auditory information in the ‘short auditory store’ typically decays within 200-300ms, whereas auditory information in the ‘long auditory store’ is thought to be preserved for several seconds and can be replayed (see Crowder, 1993; Kallman & Massaro, 1979). In the Colavita task, the auditory information in the short auditory store would be used when participants immediately respond to the auditory stimulus. The existence of the long term form of echoic memory raises the possibility that participants in the Colavita task would be able to store the auditory stimulus in the auditory buffer and therefore postpone their processing of the auditory target until after they had responded to the visual stimulus. Thus, the potential use of echoic memory by participants could represent another factor that may have contributed to the slower RTs to auditory than to visual stimuli on the bimodal trials. In sum, an explanation has been put forward to describe how participants successfully responded to both the auditory and visual components of a bimodal stimulus in the Colavita task. If participants had a delayed perception of the auditory stimulus (presumably as a result of their attending toward the visual modality), they would still be able to respond to both components of the bimodal stimulus by either delaying their auditory response, delaying their visual response in order to make a response couplet, or by replaying the auditory stimulus in their auditory echoic memory (after processing of the visual stimulus had been completed). In addition, participants would also respond to the auditory component of the bimodal stimulus if 236

CHAPTER 9 - DISCUSSION

they had endogenously attended toward the auditory modality (however, this seemed to occur rarely), if the participants replayed the auditory stimulus in their echoic memory, if the temporal separation between the auditory and visual stimuli was great enough that participants could respond to the two stimuli as if they were sequentially presented unimodal stimuli, or if a redundant spatial cue alerted participants to the fact that two stimuli had in fact been presented.

9.3

THE COLAVITA EFFECT & EXTINCTION
One reason for studying the Colavita effect is because of its apparent

similarity to the clinical phenomenon of extinction which is observed in certain neuropsychological patients (e.g., Bender, 1952; Frassinetti et al., 2002; Mattingley et al., 1997; Rapp & Hendel, 2003; Sarri et al., 2006; see Maravita et al., 2001; Làdavas & Farnè, 2004, for reviews of the literature on crossmodal extinction; also see Chapter 1, Section 1.7). Patients suffering from crossmodal extinction are typically able to detect a single stimulus presented to either the ipsi- or contralesional side of space, but fail to report the same stimulus when it is presented on the contralesional side of space while a concurrent stimulus (in another modality) is presented on the ipsilesional side. Likewise, in the Colavita effect, participants are able to detect an auditory stimulus when it is presented alone, but sometimes fail to report the auditory stimulus when it is presented concurrently with a visual stimulus. Thus, given the similarity (at least, at a superficial level) between the two phenomena, the Colavita effect could possibly be thought of as a non-pathological form of crossmodal extinction which may eventually provide a model of crossmodal extinction in patients.

237

CHAPTER 9 - DISCUSSION

There are, however, certain notable differences between the phenomenon of crossmodal extinction and the Colavita effect; first, crossmodal extinction has only been reported between touch and vision (e.g., Bueti et al., 2004; di Pellegrino et al., 1997b; Farnè, Pavani, Meneghello, & Làdavas, 2000; Làdavas, di Pellegrino, Farnè, & Zeloni, 1998; Làdavas, Zeloni, & Farnè, 1998; Maravita, Spence, Clarke, Husain, & Driver, 2000), or between touch and audition (e.g., Farnè & Làdavas, 2002; Làdavas, Pavani, & Farnè, 2001), but never between audition and vision (see reviews on crossmodal extinction documenting this surprising absence of audiovisual extinction; e.g., Làdavas & Farnè, 2004; Maravita et al., 2001). In contrast, the experiments reported in this thesis have focused on the Colavita effect that emerges between audition and vision. Note, however, that the Colavita effect has recently been demonstrated between vision and touch as well (Hartcher-O’Brien, Gallace, Krings, Koppen, & Spence, 2008). Second, it is the ipsilesional stimulus that extinguishes the contralesional stimulus in crossmodal extinction (i.e., the stimulus presented on the patient’s right hand side extinguishes the stimulus presented on the left hand side and not vice versa; though see Costantini et al., 2007, for an exception, as will be discussed later), whereas the side of space from which the stimuli are presented does not appear to affect the magnitude of the Colavita effect (see Chapter 6, Section 6.1.2). Given the differences between the phenomena of crossmodal extinction and the Colavita effect, as well as the possibility that the Colavita effect could potentially provide valuable insights to the study of extinction, it is worthwhile comparing the two phenomena to ascertain whether there is merely a surface similarity between them or whether instead there is a structural similarity between them. Comparisons will now be drawn between the role that exogenous and endogenous attention, spatial and

238

CHAPTER 9 - DISCUSSION

temporal coincidence, and perceptual and post-perceptual processes, play in the audiovisual Colavita effect, and the role that they play in crossmodal extinction. Table 9.1 provides a summary of these comparisons.

9.3.1 Attention Recently, researchers investigating the phenomenon of extinction have begun to turn their efforts toward understanding the contributions of endogenous and exogenous orienting of attention to extinction. The neuropsychological evidence that has emerged thus far suggests that the basic mechanism leading to extinction is an impairment in exogenous orienting toward the contralesional side of space (e.g., Bartolomeo, Siéroff, Decaix, & Chokron, 2001; see Bartolomeo & Chokron, 2002; O’Regan & Nöe, 2001, for a review). In particular, Bartolomeo and Chokron have posited that extinction is caused by patients’ tendency to have their attention more readily captured exogenously by stimuli presented on the dominant side of space than on the other non-dominant side. In a similar vein, one of the arguments made in this thesis is that the Colavita effect is caused, at least in part, by participants’ tendency to have their attention more readily captured exogenously by the dominant modality (i.e., vision) than by the non-dominant modality (i.e., audition). Thus, it appears that the ability of the dominant stimulus (where dominance is determined by the side of presentation for extinction patients, and determined by dominant modality for the Colavita effect) to capture attention exogenously could potentially play a similar role in both extinction and in the Colavita effect. Interestingly, studies of crossmodal extinction (between touch and vision) often report that ipsilesional visual stimuli appear to extinguish the perception of contralesional tactile stimuli more readily than ipsilesional tactile stimuli can 239

CHAPTER 9 - DISCUSSION

extinguish contralesional visual stimuli (e.g., Costantini et al., 2007). Costantini et al. even demonstrated that a contralesional visual stimulus can extinguish a contralesional tactile stimulus. Studies investigating exogenous attentional capture in normal participants have shown that visual stimuli are more effective at capturing attention exogenously than are tactile stimuli (e.g., Turatto et al., 2004). Taken together, these findings therefore suggest that crossmodal extinction, like the Colavita effect, is also modulated by which stimulus is more effective at exogenously capturing a patient’s/participant’s attention (also see Costantini et al., 2007, on this point).
Table 9.1. Table summarizing the similarities and differences between the phenomena of crossmodal extinction and the Colavita effect. The table also highlights the comparison between the factors that contribute to the Colavita effect and those contributing to the phenomenon of crossmodal extinction.
Crossmodal extinction Dominant stimulus Non-dominant stimulus Factors that contribute to, or modulate, the phenomenon The relative spatial positions of the stimuli Temporal separation between the stimuli Prior entry of the dominant stimulus Exogenous attention Endogenous attention Perceptual processes (a decrease in sensitivity to the non-dominant stimulus in the presence of the dominant stimulus) Decisional processes (a decrease in the criterion for reporting the dominant stimulus) Visual/Auditory Tactile Yes Yes Yes Yes Yes Yes Colavita effect Visual Auditory/Tactile Yes12 Yes Yes Yes Yes Yes

Yes

Yes (in the early studies of the Colavita effect) No (in the experiments reported in this thesis

In addition, studies of extinction have shown that although patients with extinction can direct their attention to the contralesional side when required, a bias that extinction patients have to attend to the ipsilesional side of space contributes to the emergence of extinction (e.g., Karnath, 1988; Mattingley, Bradshaw, Bradshaw, & Nettleton, 1994). This is paralleled by the finding in this thesis that participants’ tendency to attend to the visual modality contributes to the Colavita effect. This

12

Note, however, that the Colavita effect and extinction are affected by the relative spatial positions of the stimuli in somewhat different ways.

240

CHAPTER 9 - DISCUSSION

therefore suggests that crossmodal extinction, like the Colavita effect, is also modulated by which stimulus patients/participants tend to endogenously attend to.

9.3.2 Spatial and temporal separation The Colavita effect and the phenomenon of extinction are both modulated by the temporal separation between the stimuli; the Colavita effect is eliminated when the auditory and visual stimuli are presented far enough apart so that participants can respond to them sequentially (as demonstrated in Chapter 5), and the extent to which the ipsilesional stimulus extinguishes the contralesional stimulus in extinction diminishes the further apart the stimuli are in time (Baylis et al., 2002; Bueti et al., 2004; Costantini et al., 2007; di Pellegrino et al., 1997a; Guerrini et al., 2003; Karnath et al., 2002; Rorden et al., 1997). As in the results reported in Chapter 5, extinction tends to be eliminated at SOAs larger than 150ms. Meanwhile, although both the Colavita effect and crossmodal extinction are affected by the relative spatial positions of the stimuli, they are affected by the relative spatial positions of the stimuli in different ways. The Colavita effect is modulated by whether or not the stimuli are presented from the same location (being larger when they are presented from the same location than when they are presented from different locations), and the visual stimuli will dominate regardless of whether the stimuli are presented from the same or different positions, and regardless of which sides the stimuli are presented on. In contrast, in cases of crossmodal extinction it is only the ipsilesional stimulus that extinguishes the contralesional stimulus, and crossmodal extinction does not tend to occur when the stimuli are presented from the same position (see Làdavas & Farnè, 2004, for a review; though see Costantini et al., 2007, for an exception). The ipsilesional stimulus will extinguish the contralesional 241

CHAPTER 9 - DISCUSSION

stimulus regardless of the stimulus modalities involved (i.e., an ipsilesional visual stimulus will extinguish a contralesional tactile stimulus, and an ipsilesional tactile stimulus will extinguish a contralesional visual stimulus; e.g., Mattingley et al., 1997). In sum, therefore, the temporal separation between the stimuli and their relative spatial positions contribute to both the Colavita effect and to the clinical phenomenon of extinction. However, the Colavita effect and extinction are affected by the relative spatial positions of the stimuli in different ways.

9.3.3 Perceptual and post-perceptual processes Recent research has shown that clinical extinction can reflect a decrease in a patient’s sensitivity to the contralesional stimulus and a shift in their response criterion where patients respond more conservatively to stimuli that they perceive as less intense (Gorea & Sagi, 2000, 2002; Olson, Stark, & Chatterjee, 2003; Ricci & Chatterjee, 2004; Sarri et al., 2006). In other words, extinction appears to reflect both perceptual and decisional processes. In contrast, the results of the experiment reported in Chapter 8 suggested that the Colavita effect (at least the one that emerges when the stimuli are presented at their 75% detection threshold) reflects only a perceptual, but not a decisional, phenomenon; where participants showed a decrease in sensitivity, but no change in their response criterion, to auditory stimuli when they were presented concurrently with visual stimuli. Hence, although perceptual processes appear to contribute to both extinction and to the Colavita visual dominance effect, post-perceptual decisional factors (criterion shifting) only appear to affect extinction patients but not the participants in studies of the Colavita effect (at least when the Colavita task was conducted in the manner outlined in the experiment reported in this thesis). It is likely, however, that decisional factors may have contributed to the 242

CHAPTER 9 - DISCUSSION

Colavita effect reported in the early studies of the phenomenon (e.g., Colavita, 1974; Colavita et al., 1976; Colavita & Weisberg, 1979; Egeth & Sager, 1977). To summarize, the comparisons between the Colavita effect and the phenomenon of crossmodal extinction revealed that the orienting of

patients’/participants’ attention exogenously and endogenously, the temporal separation between the stimuli, and perceptual processes (i.e., a decrease in sensitivity toward the non-dominant stimulus when presented concurrently with a dominant stimulus) all contribute in similar ways to the two phenomena (see Table 9.1 for a summary). These similarities persist despite the fact that the phenomena are shown in different types of participants (normal participants versus patients suffering from extinction), and occur in different sensory modalities (visual-auditory versus visualtactile). As the Colavita effect observed between audition and vision appears to bear more than just a surface similarity to crossmodal extinction, it therefore provides a useful addition to the examples of natural extinction that have been demonstrated in neurologically-normal participants (for examples of within-modality extinction, see Bender et al., 1954; Gorea & Sagi, 2002). However, as visuotactile extinction is the most commonly investigated form of crossmodal extinction (see reviews of the literature, e.g., Làdavas & Farnè, 2004; Maravita et al., 2001), the insights gained by investigating the visuotactile Colavita effect may provide an even more valuable contribution to understanding crossmodal extinction. A particular advantage of studying the Colavita effect, in order to gain insights into extinction, is that the Colavita effect can be investigated in normal populations and is therefore easier to study than extinction, which occurs relatively more rarely in patients, and is often accompanied by a variety of other deficits that change over time.

243

CHAPTER 9 - DISCUSSION

9.4

CONCLUSIONS
The aim of the experiments reported in this thesis was to investigate some of

the key factors that contribute to the Colavita effect in order to gain a deeper understanding of the mechanisms causing it. The findings presented in this thesis advance several conclusions that may be important to the field of multisensory integration, in particular to the area of visual dominance, and also provide a potential springboard for the study of crossmodal extinction in normal participants. The main findings reported in this thesis were that the Colavita effect is modulated by the modality to which participants endogenously and/or exogenously attend and the stimulus that they perceived first. An explanation describing the Colavita effect was put forward which suggested that participants’ tendency to have their attention directed (either endogenously and/or exogenously) toward the visual modality may lead to the prior entry of the visual stimulus, which in turn may result in the auditory stimulus arriving during the attentional dwell time of the visual stimulus. Finally, the audiovisual Colavita effect and the phenomenon of crossmodal extinction are very similar (not only in terms of their surface qualities, but also in terms of the factors that affect them). The study of the visuotactile Colavita effect may therefore hold great promise in enlightening the study of crossmodal extinction.

9.5

SUGGESTIONS FOR FUTURE RESEARCH
One of the most important next steps for future research on the audiovisual

Colavita effect would be to investigate its neural basis (in terms of the auditory stimulus not being perceived in the presence of a visual stimulus). One could, for example, use functional magnetic resonance imaging (fMRI) to compare the pattern 244

CHAPTER 9 - DISCUSSION

of brain activation on those trials in which the auditory component of a bimodal stimulus is detected versus when it is not (i.e., when the Colavita visual dominance effect occurs). An investigation is currently underway (Scott Sinnett, 15th June, 2007, personal communication) to explore the electrophysiological activity (ERPs) related to the Colavita effect. In particular, Sinnett and his colleagues are measuring the electrical activity evoked by the presentation of unimodal auditory, unimodal visual, and bimodal stimuli in order to determine whether there are changes in evoked activation for the auditory/visual stimulus on those bimodal trials in which the Colavita effect occurs versus when both the auditory and visual stimuli are detected. Based on the results reported in this thesis suggesting that shifts in sensitivity toward the auditory stimulus contribute to the effect, it could be predicted that there would be a decreased evoked activation for the auditory stimulus on bimodal trials in which the Colavita error occurs. Based on findings in the AB literature that successful detection of a T2 target is contingent on the elicitation of a P3 component (Sergent et al., 2005), one might also predict that the detection of the auditory component of the bimodal stimulus might similarly be contingent upon the elicitation of a P3 component. It would also be interesting to investigate the modality combinations in which the Colavita effect occurs. For example, one could investigate whether there is an audiotactile Colavita effect, and answer the question of which modality would dominate. Given that studies of audiotactile extinction have revealed that an ipsilesional auditory stimulus more readily extinguishes a contralesional tactile stimulus than vice versa, one would predict that the auditory stimulus would dominate. It would also be worthwhile extending the work of Shapiro et al. (1984) to investigate (using a Colavita paradigm free from response biases, etc) the effects of arousal on visual dominance. In particular, one could explore whether the Colavita

245

CHAPTER 9 - DISCUSSION

visual dominance effect would still occur if a visual stimulus were to be paired with a painful stimulus (such as a painful laser stimulus; cf. Shapiro et al., 1984; Spence, Bentley, Phillips, McGlone, & Jones, 2002), given that painful stimuli are particularly effective at capturing attention exogenously (Koster, Crombez, Van Damme, Verschuere, & De Houwer, 2005; Legrain, Bruyer, Guérit, & Plaghki, 2005; Van Damme, Crombez, Hermans, Koster, & Eccleston, 2006). Finally, as indicated earlier, what would really increase the value of researching the Colavita effect would be to investigate the visuotactile Colavita effect in greater depth. An interesting study to conduct would involve using transcranial magnetic stimulation (TMS; which creates a ‘virtual’ lesion on the area of the brain to which TMS is applied) to investigate whether applying TMS to the typical site of lesions in extinction patients (the right parietal cortex; e.g., Vallar, 1993) would give rise to an increased visuotactile Colavita effect in normal participants. Studies investigating the links between the visuotactile Colavita effect and extinction would potentially provide a great deal of insight into the phenomenon of visuotactile crossmodal extinction by providing a model in normal participants of crossmodal extinction. Then, we could really begin to start ‘seeing the light’.

246

BIBLIOGRAPHY

BIBLIOGRAPHY
Ades, H. W. (1944). Midbrain auditory mechanisms in cats. Journal of Neurophysiology, 7, 415-424. Allen, P. G., & Kolers, P. A. (1981). Sensory specificity of apparent motion. Journal of Experimental Psychology: Human Perception & Performance, 7, 13181326. Alpern, M. (1954). The relation of visual latency to intensity. A.M.A. Archives of Ophthalmology, 142, 258-267. Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience, 3, 324-330. Amedi, A., von Kriegstein, K., van Atteveldt, N. M., Beauchamp, M. S., & Naumer, M. J. (2005). Functional imaging of human crossmodal identification and object recognition. Experimental Brain Research, 166, 559-571. Anstis, S. M. (1973). Hearing with the hands. Perception, 2, 337-341. Aristotle (1977). De Anima (On the Soul), translated by H. Lawson-Tancred. London: Penguin Books.

247

BIBLIOGRAPHY

Arnell, K., & Duncan, J. (2002). Separate and shared sources of dual-task cost in stimulus identification and response selection. Cognitive Psychology, 44, 105147. Arnell, K. M., & Jenkins, R. (2004). Revisiting within modality and cross-modality attentional blinks: Effects of target-distractor similarity. Perception & Psychophysics, 66, 1147-1161. Arnell, K., & Jolicoeur, P. (1999). The attentional blink across stimulus modalities: Evidence for central process limitations. Journal of Experimental Psychology: Human Perception & Performance, 25, 630-648. Arnell, K. M., & Larson, J. M. (2002). Cross-modality attentional blink without preparatory task-set switching. Psychonomic Bulletin & Review, 9, 497-506. Arvidson, P. S. (2003). A lexicon of attention: from cognitive science to phenomenology. Phenomenology and the Cognitive Sciences, 2, 99-132. Atkins, J. E., Fiser, J., & Jacobs, R. A. (2001). Experience-dependent visual cue integration based on consistencies between visual and haptic percepts. Vision Research, 41, 449-461. Aschersleben, G., Bachmann, T., & Műssler, J. (Eds.). (1999). Cognitive contributions to the perception of spatial and temporal events (Advances in Psychology, Vol. 129). North-Holland: Elsevier Science B.V. Bartolomeo, P., & Chokron, S. (2002). Orienting of attention in left unilateral neglect. Neuroscience and Biobehavioral Reviews, 26, 217-234. Bartolomeo, P., Siéroff, E., Decaix, C., & Chokron, S. (2001). Modulating the attentional bias in unilateral neglect: The effects of the strategic set. Experimental Brain Research, 137, 424-431.

248

BIBLIOGRAPHY

Basil, M. D. (1994). Multiple resource theory I: Application to television viewing. Communication Research, 21, 177-207. Baylis, G. C., Driver, J., & Rafal, R. D. (1993). Visual extinction and stimulus repetition. Journal of Cognitive Neuroscience, 5, 453-466. Baylis, G. C., Simon, S. L., Baylis, L. L., & Rorden, C. (2002). Visual extinction with double simultaneous stimulation: What is simultaneous? Neuropsychologia, 40, 1027-1034. Bedford, F. L. (2001). Towards a general law of object/numerical identity. Current Psychology of Cognition, 20, 113-175. Ben-Artzi, E., & Marks, L. E. (1995). Visual-auditory interaction in speeded classification: Role of stimulus difference. Perception & Psychophysics, 57, 1151-1162. Bender, M. B. (1952). Disorders in perception. Springfield, IL: Charles Thomas. Bender, M. B., Green, M. A., & Fink, M. (1954). Patterns of perceptual organization with simultaneous stimuli. Archives of Neurology and Psychiatry, 72, 233255. Bensmaïa, S., Killebrew, J., & Craig, J. C. (2006). The influence of visual motion on tactile motion perception. Journal of Neurophysiology, 96, 1625-1637. Bernstein, L. E., Auer, E. T., Moore, J. K., Ponton, C. W., Don, M, & Singh, M. (2002). Visual speech perception without primary auditory cortex activation. Neuroreport, 13, 311-315. Bertelson, P. (1961). Sequential redundancy and speed in a serial two-choice responding task. Quarterly Journal of Experimental Psychology, 13, 90-102. Bertelson, P. (1998). Starting from the ventriloquist: The perception of multimodal events. In M. Sabourin, F. I. M. Craik, & M. Robert (Eds.), Advances in 249

BIBLIOGRAPHY

psychological science, Vol. 2: Biological and cognitive aspects (pp. 419-439). Hove, UK: Psychological Press. Bertelson, P., & Aschersleben, G. (1998). Automatic visual bias of perceived auditory location. Psychonomic Bulletin & Review, 5, 482-489. Bertelson, P., & Aschersleben, G. (2003). Temporal ventriloquism: Crossmodal interaction on the time dimension. 1. Evidence from time order judgments. International Journal of Psychophysiology, 50, 147-155. Bertelson, P., & de Gelder, B. (2004). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 141-177). Oxford, UK: Oxford University Press. Bertelson, P., & Radeau, M. (1976). Ventriloquism, sensory interaction, and response bias: Remarks on the paper by Chloe, Welch, Gilford, and Juola. Perception & Psychophysics, 19, 531-535. Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception & Psychophysics, 29, 578-584. Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321-332. Bertelson, P., Vroomen, J., Wiegeraad, G., & de Gelder, B. (1994). Exploring the relation between McGurk interference and ventriloquism. Proceedings of the 1994 International Conference on Spoken Language Processing, 2, 559-562. Bland., J. M., & Altman, D. G. (1995, Feb). Calculating correlation coefficients with repeated measures observations: Part 1 - correlation within subjects. British Medical Journal, 310, 446.

250

BIBLIOGRAPHY

Broadbent, D. E., & Broadbent, M. H. P. (1987). From detection to identification: Response to multiple targets in rapid serial visual presentation. Perception & Psychophysics, 42, 105-113. Bronkhorst, A. W., van der Hoeven, M., Theeuwes, J., van der Burg, E., & Koelewijn, T. (2006). Factors influencing the auditory and cross-modal attentional blink. Journal of the Acoustical Society of America, 120, 3128. Büchel, C., Josephs, O., Rees, G., Turner, R., Frith, C. D., & Friston, K. J. (1998). The functional anatomy of attention to visual motion. A functional MRI study. Brain, 121, 1281-1294. Bueti, D., Costantini, M., Forster, B., & Aglioti, S. (2004). Uni- and cross-modal temporal modulation of tactile extinction in right brain damaged patients. Neuropsychologia, 42, 1689-1696. Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C., McGuire, P. K., Woodruff, P. W., Iversen, S. D., & David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276, 593-596. Calvert G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. NeuroReport, 10, 2619-23. Calvert, G. A., Campbell, R., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex. Current Biology, 10, 649-657. Calvert, G. A., Iversen, S. D., & Brammer, M. J. (2000). Calvert, G. A., Spence, C., & Stein, B. E. (Eds.) (2004). The handbook of multisensory processes. Cambridge, MA: MIT Press.

251

BIBLIOGRAPHY

Campbell, K. C., & Proctor, R. W. (1993). Repetition effects with categorizable stimulus and response sets. Journal of Experimental Psychology: Learning, Memory, & Cognition, 19, 1345-1362. Cate, A., & Behrmann, M. (2002). Spatial and temporal influences on extinction. Neuropsychologia, 40, 2206-2225. Chan, A. H. S., & Chan, K. W. L. (2006). Synchronous and asynchronous presentations of auditory and visual signals: Implications for control console design. Applied Ergonomics, 37, 131-140. Chelazzi, L., Marzi, C. A., Panozzo, G., Pasqualini, N., Tassinari, G., & Tomazzoli, L. (1998). Hemiretinal differences in speed of light detection in esotropic amblyopes. Vision Research, 28, 95-104. Choe, C. S., Welch, R. B., Gilford, R. M., & Juola, J. F. (1975). The 'ventriloquist effect': Visual dominance or response bias? Perception & Psychophysics, 18, 55-60. Chun, M. M., & Potter, M. C. (1995). A two-stage model for multiple target detection in rapid serial visual presentation. Journal of Experimental Psychology: Human Perception & Performance, 21, 109-127. Chun, M. M., & Wolfe, J. (2001). Visual attention. In E. B. Goldstein (Ed.), Blackwell’s handbook of perception (pp. 272-310). Oxford, UK: Blackwell. Cochran, W. G. (1957). Analysis of covariance: Its nature and uses. Biometrics, 13, 261-81. Colavita, F. B. (1974). Human sensory dominance. Perception & Psychophysics, 16, 409-412. Colavita, F. B., Tomko, R., & Weisberg, D. (1976). Visual prepotency and eye orientation. Bulletin of the Psychonomic Society, 8, 25-26. 252

BIBLIOGRAPHY

Colavita, F. B., & Weisberg, D. (1979). A further investigation of visual dominance. Perception & Psychophysics, 25, 345-347. Colin, C., Radeau, M., Deltenre, P., & Morais, J. (2001). Rules of intersensory integration in spatial scene analysis and speechreading. Psychologica Belgica, 41, 131-144. Coren, S., Ward, L. M., & Enns, J. T. (2004). Sensation & perception (6th Ed.). Fort Worth: Harcourt Brace. Cornsweet, T. N. (1962). The staircase-method in psychophysics. American Journal of Psychology, 75, 485-491. Costantini, M., Bueti, D., Pazzaglia, M., & Aglioti, S. (2007). Temporal dynamics of visuo-tactile extinction within and between hemispaces. Neuropsychology, 21, 242-250. Cowan, N. (1984). On short and long auditory stores. Psychological Bulletin, 96, 341370. Craig, J. C. (2006). Visual motion interferes with tactile motion perception. Perception, 35, 351-367. Craig, J. C., & Baihua, X. (1990). Temporal order and tactile patterns. Perception & Psychophysics, 47, 22-34. Crowder, R. G. (1993). Auditory memory. In S. McAdams & E. Bigand (Eds.), Thinking in sound: The psychology of human audition (pp. 113-145). Oxford, UK: Oxford University Press. de Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences, 7, 460-467. di Pellegrino, G., Basso, G., & Frassinetti, F. (1997a). Spatial extinction on double asynchronous stimulation. Neuropsychologia, 35, 1215-1223. 253

BIBLIOGRAPHY

di Pellegrino, G., Làdavas, E., & Farné, A. (1997b). Seeing where your hands are. Nature, 388, 730. Dixon, N. F., & Spitz, L. (1980). The detection of auditory visual desynchrony. Perception, 9, 719-721. Driver, J., & Spence, C. (2000). Multisensory perception: Beyond modularity and convergence. Current Biology, 10, R731-R735. Driver, J., & Spence, C. (2004). Crossmodal spatial attention: Evidence from human performance. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 179-220). Oxford, UK: Oxford University Press. Duncan, J., Ward, R., & Shapiro, K. (1994). Direct measurement of attentional dwell time in human vision. Nature, 369, 313-315. Efron, R. (1963). Temporal perception, aphasia, and déjà vu. Brain, 86, 403-424. Egeth, H. E., & Sager, L. C. (1977). On the locus of visual dominance. Perception & Psychophysics, 22, 77-86. Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429-433. Ernst, M. O., Banks, M. S., & Bülthoff, H. H. (2000). Touch can change visual slant perception. Nature Neuroscience, 3, 69-73. Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends in Cognitive Sciences, 8, 162-169. Fagot, C., & Pashler, H. (1992). Making two responses to a single object: Implications for the central attentional bottleneck. Journal of Experimental Psychology: Human Perception & Performance, 18, 1058-1079.

254

BIBLIOGRAPHY

Farnè, A., & Làdavas, E. (2002). Auditory peripersonal space in humans. Journal of Cognitive Neuroscience, 14, 1030-1043. Farnè, A., Pavani, F., Meneghello, F., & Làdavas, E. (2000). Left tactile extinction following visual stimulation of a rubber hand. Brain, 123, 2350-2360. Fendrich, R., & Corballis, P. M. (2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 719-725. Finney, D. J. (1964). Probit analysis: Statistical treatment of the sigmoid response curve. London: Cambridge University Press. Fisher, R. A. (1932). Statistical Methods for Research Workers. Edinburgh: Oliver and Boyd. Fisher, B. D., & Pylyshyn, Z. W. (1994). The cognitive architecture of bimodal event perception: A commentary and addendum to Radeau (1994). Current Psychology of Cognition, 13, 92-96. Foree, D. D., & LoLordo, V. M. J. (1973). Attention in the pigeon: Differential effects of food-getting versus shock-avoidance procedures. Journal of Comparative & Physiological Psychology, 85, 551-558. Fort, A., Delpuech, C., Pernier, J., & Giard, M.-H. (2002). Dynamics of corticosubcortical cross-modal operations involved in audio-visual object detection in humans. Cerebral Cortex, 12, 1031-1039. Frassinetti, F., Bolognini, N., & Làdavas, E. (2002). Enhancement of visual perception by crossmodal visuo-auditory interaction. Experimental Brain Research, 147, 332-343. Frassinetti, F., Pavani, F., & Làdavas, E. (2002). Acoustical vision of neglected stimuli! Interaction among spatially converging audio-visual inputs in neglect patients. Journal of Cognitive Neuroscience, 14, 62-69. 255

BIBLIOGRAPHY

Freides, D. (1974). Human information processing and sensory modality: Crossmodal functions, information complexity, memory, and deficit. Psychological Bulletin, 81, 284-310. Frick, R. W. (1995). Accepting the null hypothesis. Memory & Cognition, 23, 132138. Gaiman, N. (2001). American Gods. London: Headline Publishing. Gallace, A., Tan, H. Z., & Spence, C. (2007). The body surface as a communication system: The state of the art after 50 years. Presence: Teleoperators & Visual Environments, 16, 655-676. Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11, 473-490. Gibson, J. J. (1943). Adaptation, after-effect and contrast in the perception of curved lines. Journal of Experimental Psychology, 16, 1-31. Giesbrecht, B., & Di Lollo, V. (1998). Beyond the attentional blink: Visual masking by object substitution. Journal of Experimental Psychology: Human Perception & Performance, 24, 1454-1466. Goldring. J., Dorris, M., Corneil, B., Balantyne, P., & Munoz, D. (1996). Combined eye-head gaze shifts to visual and auditory targets in humans. Experimental Brain Research, 111, 68-78. Gorea, A., & Sagi, D. (2000). Failure to handle more than one internal representation in visual detection tasks. Proceedings of the National Academy USA, 97, 12380-12384. Gorea, A., & Sagi, D. (2002). Natural extinction: A criterion shift phenomenon. Visual Cognition, 9, 913-936. 256

BIBLIOGRAPHY

Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley. Guerrini, C., Berlucchi, G., Bricolo, E., & Aglioti, S. M. (2003). Temporal modulation of spatial tactile extinction in right-brain-damaged patients. Journal of Cognitive Neuroscience, 15, 523-536. Guest, S., & Spence, C. (2003). Tactile dominance in the speeded discrimination of textures. Experimental Brain Research, 150, 201-207. Hall, K. R. L., & Earle, A. E. (1954). A further study of the pendulum phenomenon. Quarterly Journal of Experimental Psychology, 6, 112-124. Hamlin, A. J. (1895). On the least observable interval between stimuli addressed to disparate senses and to different organs of the same sense. American Journal of Psychology, 6, 564-575. Ho, C., Reed, N., & Spence, C. (2007). Multisensory in-car warning signals for collision avoidance. Human Factors, 49, 1107-1114. Ho, C., & Spence, C. (2005). Assessing the effectiveness of various auditory cues in capturing a driver's visual attention. Journal of Experimental Psychology: Applied, 11, 157-174. Hohnsbein, J., Falkenstein, M., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. I. Simple and choice reaction tasks. Electroencephalography and Clinical Neurophysiology, 78, 438-446. Holender, D. (1980). Interference between a vocal and a manual response to the same stimulus. In G. E. Stelmach, & J. Requin, (Eds.), Tutorials in motor behaviour (pp. 421-431). Amsterdam: North-Holland.

257

BIBLIOGRAPHY

Howard, I. P., & Templeton, W. B. (1966). Human spatial orientation. New York: Wiley. ITU-R BT.1359-1 (1998). Relative timing of sound and vision for broadcasting. Question ITU-T 35/11. Jackson, C. V. (1953). Visual factors in auditory localization. Quarterly Journal of Experimental Psychology, 5, 52-65. James, W. (1890). Principles of psychology (Vol. 1). New York: Henry Holt. Jäncke, L., Mirzazadeb, S., & Shah, N. J. (1999). Attention modulates activity in the primary and the secondary auditory cortex: A functional magnetic resonance imaging. Neuroscience Letters, 266, 125-128. Jaśkowski, P. (1996). Simple reaction time and perception of temporal order: Dissociations and hypotheses. Perceptual & Motor Skills, 82, 707-730. Jaśkowski, P. (1999). Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations. In G. Aschersleben, T. Bachmann, & J. Műssler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 265-282). North-Holland: Elsevier Science B.V. Johnson, T. L., & Shapiro, K. L. (1989). Attention to auditory and peripheral visual stimuli: Effects of arousal and predictability. Acta Psychologica, 72, 233-245. Jolicoeur, P. (1998). Modulation of the attentional blink by on-line response selection: Evidence from speeded and unspeeded task 1 decisions. Memory & Cognition, 26, 1014-1032. Jolicoeur, P. (1999). Restricted attentional capacity between sensory modalities. Psychonomic Bulletin & Review, 6, 87-92.

258

BIBLIOGRAPHY

Jolicoeur, P., & Dell'Acqua, R. (1998). The demonstration of short-term consolidation. Cognitive Psychology, 36, 138-202. Jolicoeur, P., & Dell'Acqua, R. (1999). Attentional and structural constraints on visual encoding. Psychological Research, 62, 154-164. Jolicoeur, P., Dell'Acqua, R., & Crebolder, J. (2000). Multitasking performance deficits: Forging some links between the attentional blink and the psychological refractory period. In S. Monsell and J. Driver (Eds.), Attention and performance, XVIII: Control of cognitive processes (pp. 309-330). Cambridge, MA: MIT Press. Jones, J. A., & Munhall, K. G. (1997). The effects of separating auditory and visual sources on audiovisual integration of speech. Canadian Acoustics, 25, 13-19. Kallman, H. J., & Massaro, D. W. (1979). Similarity effects in backward recognition masking. Journal of Experimental Psychology: Human Perception & Performance, 5, 110-128. Karnath, H. O. (1988). Deficits in attention in acute and recovered visual hemineglect. Neuropsychologia, 26, 27-43. Karnath, H. O., Himmelback, M., & Rorden, C. (2002). The subcortical anatomy of human spatial neglect: Putamen, caudate nucleus and pulvinar. Brain, 125, 350-360. Keetels, M., & Vroomen, J. (2005). The role of spatial disparity and hemifields in audio-visual temporal order judgments. Experimental Brain Research, 167, 635-640. King, A. J. (2005). Multisensory integration: Strategies for synchronization. Current Biology, 15, R339-R341.

259

BIBLIOGRAPHY

Kitagawa, N., & Ichihara, S. (2002). Hearing visual motion in depth. Nature, 416, 172-174. Kitterle, F. L. (1986). Psychophysics of lateral tachistoscopic presentation. Brain & Cognition, 5, 131-162. Klein, R. M. (1977). Attention and visual dominance: A chronometric analysis. Journal of Experimental Psychology: Human Perception & Performance, 3, 365-378. Knudsen, E. I., & Brainard, M. S. (1995). Creating a unified representation of visual and auditory space in the brain. Annual Review of Neuroscience, 18, 19-43. Kornblum, S. (1973). Sequential effects in choice reaction time: A tutorial review. In S. Kornblum (Ed.), Attention and performance (Vol. 4, pp. 259-288). New York: Academic Press. Koster, E. H. W., Crombez, G., Van Damme, S., Verschuere, B., & De Houwer, J. (2005). Signals for threat modulate attentional capture and holding: Fearconditioning and extinction during the exogenous cueing task. Cognition & Emotion, 19, 771-780. Làdavas, E., Carletti, M., & Gori, G. (1994). Automatic and voluntary orienting of attention in patients with visual neglect: Horizontal and vertical dimensions. Neuropsychologia, 32, 1195-1208. Làdavas, E., di Pellegrino, G., Farnè, A., & Zeloni, G. (1998). Neuropsychological evidence of an integrated visuotactile representation of peripersonal space in humans. Journal of Cognitive Neuroscience, 10, 581-589. Làdavas, E., & Farnè, A. (2004). Neuropsychological evidence for multimodal representations of space near specific body parts. In C. Spence & J. Driver

260

BIBLIOGRAPHY

(Eds.), Crossmodal space and crossmodal attention (pp. 69-98). Oxford, UK: Oxford University Press. Làdavas, E., Pavani, F., & Farnè, A. (2001). Auditory peripersonal space in humans: A case of auditory-tactile extinction. Neurocase, 7, 97-103. Làdavas, E., Zeloni, G., & Farne, A. (1998b). Visual peripersonal space centred on the face in humans. Brain, 121, 2317-2326. Laurienti, P. J., Kraft, R. A., Maldjian, J. A., Burdette, J. H., & Wallace, M. T. (2004). Semantic congruence is a critical factor in multisensory behavioral performance. Experimental Brain Research, 158, 405-414. Lavie, N. (2005). Distracted and confused?: Selective attention under load. Trends in Cognitive Sciences, 9, 75-82. Lederman. S. J., Thorne, G., & Jones, B. (1986). Perception of texture by vision and touch: Multidimensionality and intersensory integration. Journal of

Experimental Psychology: Human Perception & Performance, 12, 169-180. Legrain, V., Bruyer, R., Guérit, J.-M., & Plaghki, L. (2005). Involuntary orientation of attention to unattended deviant nociceptive stimuli is modulated by concomitant visual task difficulty: Evidence from laser-evoked potentials, Clinical Neurophysiology, 116, 2165-2174. Levy, J., Pashler, H., & Boer, E. (2006). Central interference in driving: Is there any stopping the psychological refractory period? Psychological Science, 17, 228235. Lewald, J., Ehrenstein, W. H., & Guski, R. (2001). Spatio-temporal constraints for auditory-visual integration. Behavioural Brain Research, 121, 69-79. Logan, G. D. (1992). Attention and preattention in theories of automaticity. American Journal of Psychology, 105, 317-339. 261

BIBLIOGRAPHY

Lyons, G., Sanabria, D., Vatakis, A., & Spence, C. (2006). The modulation of crossmodal integration by unimodal perceptual grouping: A visuo-tactile apparent motion study. Experimental Brain Research, 174, 510-516. Macaluso, E., Frith, C., & Driver, J. (2000). Selective spatial attention in vision and touch: Unimodal and multimodal mechanisms revealed by PET. Journal of Neurophysiology, 83, 3062-3075. Macmillan, N. A., & Creelman, C. D. (1990). Detection theory: A user’s guide. Cambridge, UK: Cambridge University Press. Maki, W. S., Couture, T., Frigen, K., & Lien, D. (1997). Sources of the attentional blink during rapid serial visual presentation: Perceptual interference and retrieval competition. Journal of Experimental Psychology: Human

Perception & Performance, 23, 1393-1411. Manly, T., Robertson, I. H., Galloway, M., & Hawkins, K. (1999). The absent mind: Further investigations of sustained attention to response. Neuropsychologia, 37, 661-670. Maravita, A., Husain, M., Clarke, K., & Driver, J. (2001). Reaching with a tool extends visual-tactile interactions into far space: Evidence from cross-modal extinction. Neuropsychologia, 39, 580-585. Maravita, A., Spence, C., Clarke, K., Husain, M., & Driver, J. (2000). Vision and touch through the looking glass in a case of crossmodal extinction. Neuroreport, 11, 3521-3526. Marcell, M. E., Borella, D., Greene, M., Kerr, E., & Rogers, S. (2000). Confrontation naming of environmental sounds. Journal of Clinical Experimental Neuropsychology, 22, 830-864.

262

BIBLIOGRAPHY

Marks, L. E. (1987). On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13, 384-394. Marks, L. E. (1982). Synesthetic perception and poetic metaphor. Journal of Experimental Psychology: Human Perception and Performance, 8, 177193. Marks, L. E. (2003). Perceptual consequences of multiple sensory systems. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 85-105). Cambridge, MA: MIT Press. Marois, R. (2005). Two-timing attention. Nature Neuroscience, 8, 1285-1286. Marois, R., & Ivanoff, J. (2005). Capacity limits of information processing in the brain. Trends in Cognitive Sciences, 9, 296-305. Martens, S., Munneke, J., Smid, H. & Johnson, A. (2006). Quick minds don’t blink: Electrophysiological correlates of individual differences in attentional selection. Journal of Cognitive Neuroscience, 18, 1423-1438. Martino, G., & Marks, L. E. (1999). Perceptual and linguistic interactions in speeded classification: tests of the semantic coding hypothesis. Perception, 28, 903923. Marzi, C. A., & Stefano, M. (1971). Hemiretinal differences in visual perception. In L. Maffei (Ed.), Pathophysiology of the visual system (Vol. 30, pp. 273-278). The Hague: Dr. W. Junk Publisher. Marzi, C. A., Mancini, F., Metitieri, T., & Savazzi, S. (2005). Retinal eccentricity effects on reaction time to imagined stimuli. Neuropsychologia, 44, 14891495.

263

BIBLIOGRAPHY

Mattingley, J. B., Bradshaw, J. L., Bradshaw, J. A., & Nettleton, N. C. (1994). Residual rightward attentional bias after apparent recovery from right hemisphere damage: implications for a multicomponent model of neglect. Journal of Neurology, Neurosurgery, & Psychiatry, 57, 597-604. Mattingley, J. B., Driver, J., Beschin, N., & Robertson, I. H. (1997). Attentional competition between modalities: Extinction between touch and vision after right hemisphere damage. Neuropsychologia, 35, 867-880. McCann, R. S., & Johnston, J. C. (1992). Locus of the single-channel bottleneck in dual-task interference. Journal of Experimental Psychology: Human Perception & Performance, 18, 471-484. McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748. McRae, K., de Sa, V. R., & Seidenberg, M. S. (1997). On the nature and scope of featural representations of word meaning. Journal of Experimental Psychology: General, 126, 99-130. Melara, R. D., & O'Brien, T. P. (1987). Interactions between synesthetically corresponding dimensions. Journal of Experimental Psychology: General, 116, 323-336. Meltzer, D., & Masaki, M. A. (1973). Measures of stimulus control and stimulus dominance. Bulletin of the Psychonomic Society, 1, 28-30. Meyer, G. F., Wuerger, S. M., Röhrbein, F., & Zetzsche, C. (2005). Low-level integration of auditory and visual motion signals requires spatial colocalisation. Experimental Brain Research, 166, 538-547. Miller, J. O. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14, 247-279. 264

BIBLIOGRAPHY

Miller, J., & Schwarz, W. (2006). Dissociations between reaction times and temporal order judgments: A diffusion model approach. Journal of Experimental Psychology: Human Perception & Performance, 32, 394-412. Molholm, S., Ritter, W., Javitt, D. C., & Foxe, J. J. (2004). Multisensory visualauditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452-465. Mollon, J. D., & Perkins, A. J. (1996). Errors of judgement at Greenwich in 1796. Nature, 380, 101-102. Moore, C. M., Egeth, H., Berglan, L. R., & Luck, S. J. (1996). Are attentional dwell times inconsistent with serial visual search? Psychonomic Bulletin & Review, 3, 360-365. Morein-Zamir, S., Soto-Faraco, S., & Kingstone, A. (2003). Auditory capture of vision: Examining temporal ventriloquism. Cognitive Brain Research, 17, 154-163. Mudd, S. A. (1963). Spatial stereotypes of four dimensions of pure tone. Journal of Experimental Psychology, 66, 347-352. Mulligan, R. M., & Shaw, M. L. (1981). Attending to simple auditory and visual signals. Perception & Psychophysics, 30, 447-454. Munhall, K. G., Gribble, P., Sacco, L., & Ward, M. (1996). Temporal constraints on the McGurk effect. Perception & Psychophysics, 58, 351-362. Navarra, J., Vatakis, A., Zampini, M., Soto-Faraco, S., Humphreys, W., & Spence, C. (2005). Exposure to asynchronous audiovisual speech extends the temporal window for audiovisual integration. Cognitive Brain Research, 25, 499-507.

265

BIBLIOGRAPHY

Newell, F. N. (2004). Cross-modal object recognition. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 123-139). Cambridge, MA: MIT Press. Norman, D. A., & Shallice, T. (1980). Attention to action: Willed and automatic control of behaviour. University of California, San Diego, Center for Human Information processing: CHIP Report, 99. O’Craven, K., Rosen, B. R., Kwong, K. K., Treisman, A., & Savoy, R. L. (1997). Voluntary attention modulates fMRI activity in human MT–MST. Neuron, 18, 591-598. Odgaard, E. C., Arieh, Y., & Marks, L. E. (2003). Cross-modal enhancement of perceived brightness: Sensory interaction versus response bias. Perception & Psychophysics, 65, 123-132. Odgaard, E. C., Arieh, Y., & Marks, L. E. (2004). Brighter noise: Sensory enhancement of perceived loudness by concurrent visual stimulation. Cognitive, Affective, & Behavioral Neuroscience, 4, 127-132. Olson, I. R., Gatenby, J. C., & Gore, J. C. (2002). A comparison of bound and unbound audio–visual information processing in the human cerebral cortex. Cognitive Brain Research, 14, 129-138. Olson, E., Stark, M., & Chatterjee, A. (2003). Evidence for a unimodal somatosensory attention system. Experimental Brain Research, 151, 15-23. O’Regan, J. K., & Noë, A. (2001). A sensorimotor account of vision and visual consciousness. Behavioral & Brain Sciences, 24, 939-1011. Partan, S., & Marler, P. (1999). Communication goes multimodal. Science, 283, 12721273.

266

BIBLIOGRAPHY

Pashler, H. (1989). Dissociations and dependencies between speed and accuracy: Evidence for a two-component theory of divided attention in simple tasks. Cognitive Psychology, 21, 469-514. Pashler, H. E. (1998). The psychology of attention. Cambridge, MA: MIT Press. Pashler, H., & Baylis, G. (1991). Procedural learning: 2. Intertrial repetition effects in speeded-choice tasks. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17, 33-48. Pashler, H., & Johnston, J. C. (1989). Chronometric evidence for central postponement in temporally overlapping tasks. Quarterly Journal of Experimental Psychology, 41A, 19-45. Patching, G. R., & Quinlan, P. T. (2002). Garner and congruence effects in the speeded classification of bimodal signals. Journal of Experimental Psychology: Human Perception & Performance, 28, 755-775. Paulesu, E., Perani, D., Blasi, V., Silani, G., Borghese, A., De Giovanni, U., Sensolo, S., & Fazio, F. (2003). A Functional-Anatomical Model for Lipreading. Journal of Neurophysiology, 90, 2005-2013. Pick, H. L. Jr., Warren, D. H., & Hay, J. C. (1969). Sensory conflict in judgments of spatial direction. Perception & Psychophysics, 6, 203-205. Posner, M. I. (1978). Chronometric explorations of mind. Hillsdale, NJ: Erlbaum. Posner, M. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3-25. Posner, M. I., Nissen, M. J., & Klein, R. M. (1976). Visual dominance: An information-processing account of its origins and significance. Psychological Review, 83, 157-171.

267

BIBLIOGRAPHY

Posner, M. I., Nissen, M. J., & Ogden, W. C. (1978). Attended and unattended processing modes: The role of set for spatial location. In H. I. Pick & I. J. Saltzman (Eds.), Modes of perceiving and processing information (pp. 137157). Hillsdale, NJ: Erlbaum. Potter, M. C., Chun, M. M., Banks, B. S., & Muckenhoupt, M. (1998). Two attentional deficits in serial target search: The visual attentional blink and an amodal task-switch deficit. Journal of Experimental Psychology: Learning, Memory, & Cognition, 24, 979-992. Quinlan, P. (2000). The ‘late’ locus of visual dominance. Abstracts of the Psychonomic Society, 5, 64. Radeau, M., & Bertelson, P. (1977). Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Perception & Psychophysics, 22, 137-146. Radeau, M., & Bertelson, P. (1987). Auditory-visual interaction and the timing of inputs. Thomas (1941) revisited. Psychological Research, 49, 17-22. Randich, A., Klein, R. M., & LoLordo, V. M. (1978). Visual dominance in the pigeon. Journal of the Experimental Analysis of Behavior, 30, 129-137. Rapp, B., & Hendel, S. K. (2003). Principles of cross-modal competition: Evidence from deficits of attention. Psychonomic Bulletin & Review, 10, 210-219. Raymond, J. E., Shapiro, K. L., & Arnell, K. M. (1992). Temporary suppression of visual processing in an RSVP task: An attentional blink? Journal of Experimental Psychology: Human Perception & Psychophysics, 18, 849-860. Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078-1093.

268

BIBLIOGRAPHY

Rees, G., Frith, C., & Lavie, N. (2001). Processing of irrelevant visual motion during performance of an auditory attention task. Neuropsychologia, 39, 937-949. Reeves, B., & Voelker, D. (1993). Effects of audio-video asynchrony on viewer’s memory, evaluation of content and detection ability. Research report prepared for Pixel Instruments. Los Gatos, California. Regan, D., & Spekreijse, H. (1977). Auditory-visual interactions and the correspondence between perceived auditory space and perceived visual space. Perception, 6, 133-138. Ricci, R., & Chatterjee, A. (2004). Sensory and response contributions to visual awareness in extinction. Experimental Brain Research, 157, 85-93. Rihs, S. (1995). The influence of audio on perceived picture quality and subjective audio-visual delay tolerance. In R. Hamberg & H. de Ridder (Eds.), Proceedings of the MOSAIC workshop: Advanced methods for the evaluation of television picture quality (pp. 133-137). Eindhoven, 18-19 Sept., 1995. Risberg, A., & Lubker, J. (1978). Prosody and speech-reading. STL-Quarterly Progress and Status Report, 4, 1-16. Rock, I., & Harris, C. S. (1967, 17 May). Vision and touch. Scientific American, 216, 96-104. Rock, I., & Victor, J. (1964). Vision and touch: An experimentally created conflict between the two senses. Science, 143, 594-596. Rodway, P. (2005). The modality shift effect and the effectiveness of warning signals in different modalities. Acta Psychologica, 120, 199-226. Rorden, C., Mattingley, J. B., Karnath, H.-O., & Driver, J. (1997). Visual extinction and prior entry: Impaired perception of temporal order with intact motion perception after unilateral parietal damage. Neuropsychologia, 35, 421-433. 269

BIBLIOGRAPHY

Rosenblum, L. D., Wuestefeld, A. P., & Anderson, K. L. (1996). Auditory reachability: An affordance approach to the perception of sound source distance. Ecological Psychology, 8, 1-24. Roufs, J. A. J. (1963). Perception lag as a function of stimulus luminance. Vision Research, 3, 81-91. Ruthruff, E., Pashler, H. E., & Hazeltine, E. (2003). Dual-task interference with equal task emphasis: Graded capacity-sharing or central postponement? Perception & Psychophysics, 65, 801-816. Rutschmann, J., & Link, R. (1964). Perception of temporal order of stimuli differing in sense mode and simple reaction time. Perceptual & Motor Skills, 18, 345352. Sanford, A. J. (1971). Effects of changes in the intensity of white noise on simultaneity judgements and simple reaction time. Quarterly Journal of Experimental Psychology, 23, 296-303. Sarri, M., Blankenburg, F., & Driver, J. (2006). Neural correlates of crossmodal visual-tactile extinction and of tactile awareness revealed by fMRI in a righthemisphere stroke patient. Neuropsychologia, 44, 2398-2410. Schneider, W., Eschman, A., & Zuccolotto, A. (2002a). E-Prime user's guide. Pittsburgh: Psychology Software Tools Inc. Schneider, W., Eschman, A., & Zuccolotto, A. (2002b). E-Prime reference guide. Pittsburgh: Psychology Software Tools Inc. Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sound alters visual motion perception. Nature, 385, 308.

270

BIBLIOGRAPHY

Sergent, C., Baillet, S., & Dehaene, S. (2005). Timing of the brain events underlying access to consciousness during the attentional blink. Nature Neuroscience, 8, 1391-1400. Shallice, T. (1988). From neuropsychology to mental structure. Cambridge, UK: Cambridge University Press. Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear: Sound induced visual flashing. Nature, 408, 788. Shams, L., Kamitani, Y., & Shimojo, S. (2004). Modulations of visual perception by sound. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 27-33). Cambridge, MA: MIT Press. Shapiro, K. L., Egerman, B., & Klein, R. M. (1984). Effects of arousal on human visual dominance. Perception & Psychophysics, 35, 547-552. Shapiro, K. L., Jacobs, W. J., & LoLordo, V. M. (1980). Stimulus-reinforcer interactions in Pavlovian conditioning of pigeons: Implications for selective associations. Animal Learning & Behavior, 8, 586-594. Shapiro, K. L., Raymond, J. E., & Arnell, K. M. (1994). Attention to visual pattern information produces the attentional blink in RSVP. Journal of Experimental Psychology: Human Perception & Performance, 20, 357-371. Shibuya, S., Takahashi, T., & Kitazawa, S. (2007). Effects of visual stimuli on temporal order judgments of unimanual finger stimuli. Experimental Brain Research, 179, 709-721. Shimojo, S., & Shams, L. (2001). Sensory modalities are not separate modalities: Plasticity and interactions. Current Opinion in Neurobiology, 11, 505-509. Shomstein, S., & Yantis, S. (2004). Control of attention shifts between vision and audition in human cortex. Journal of Neuroscience, 24, 10702-10706. 271

BIBLIOGRAPHY

Shore, D. I., Spence, C., & Klein, R. M. (2001). Visual prior entry. Psychological Science, 12, 205-212. Sinnett, S., Juncadella, M., Rafal, R., & Soto-Faraco, S. (2007). A dissociation between visual and auditory hemi-inattention: Evidence from temporal order judgments. Neuropsychologia, 45, 552-560. Sinnett, S., Spence, C., Soto-Faraco, S. (2007). Visual dominance and attention: The Colavita effect revisited. Perception & Psychophysics, 69, 673-686. Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. Neuroreport, 12, 7-10. Small, D. M. (2004). Crossmodal integration - insights from the chemical senses. Trends in Neurosciences, 27, 120-123. Smith, W. F. (1933). The relative quickness of visual and auditory perception. Journal of Experimental Psychology, 16, 239-257. Snodgrass, J. G., & Vanderwart, M. (1980). A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity. Journal of Experimental Psychology: Human Learning & Memory, 6, 174-215. Soetens, E. (1998). Localizing sequential effects in serial choice reaction time with the information reduction procedure. Journal of Experimental Psychology: Human Perception & Performance, 10, 581-598. Soetens, E., Melis, A., & Notebaert, W. (2002). Sequence learning and sequential effects. Psychological Research, 69, 124-137. Soto-Faraco, S., & Kingstone, A. (2004). Multisensory integration of dynamic information. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes (pp. 49-67). Cambridge, MA: MIT Press. 272

BIBLIOGRAPHY

Soto-Faraco, S., & Spence, C. (2002). Modality-specific auditory and visual temporal processing deficits. Quarterly Journal of Experimental Psychology A, 55, 2340. Spence, C. (2007). Audiovisual multisensory integration. Acoustical Science & Technology, 28, 61-70. Spence, C., Baddeley, R., Zampini, M., James, R., & Shore, D. I. (2003). Crossmodal temporal order judgments: When two locations are better than one. Perception & Psychophysics, 65, 318-328. Spence, C., Bentley, D. E., Phillips, N., McGlone, F. P., & Jones, A. K. P. (2002). Selective attention to pain: A psychophysical investigation. Experimental Brain Research, 145, 395-402. Spence, C., & Driver, J. (1997a). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59, 1-22. Spence, C., & Driver, J. (1997b). On measuring selective attention to a specific sensory modality. Perception & Psychophysics, 59, 389-403. Spence, C., & Driver, J. (1997c). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59, 1-22. Spence, C., & Driver, J. (Eds.). (2004). Crossmodal space and crossmodal attention. Oxford, UK: Oxford University Press. Spence, C. J., & Driver, J. (1995). Spatial links in the voluntary allocation of auditory and visual attention. Australian Journal of Psychology, 47, 25. Spence, C., & Driver, J. (1996). Audiovisual links in endogenous covert spatial attention. Journal of Experimental Psychology: Human Perception and Performance, 22, 1005-1030.

273

BIBLIOGRAPHY

Spence, C., & Driver, J. (1997a). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59, 1-22. Spence, C., & Driver, J. (1997b). On measuring selective attention to a specific sensory modality. Perception & Psychophysics, 59, 389-403. Spence, C., & McDonald, J. (2004). The crossmodal consequences of the exogenous spatial orienting of attention In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processing (pp. 3-25). Cambridge, MA: MIT Press. Spence, C., Nicholls, M. E. R., & Driver, J. (2001a). The cost of expecting events in the wrong sensory modality. Perception & Psychophysics, 63, 330-336. Spence, C., Shore, D. I., & Klein, R. M. (2001b). Multisensory prior entry. Journal of Experimental Psychology: General, 130, 799-832. Spence, C., & Squire, S. B. (2003). Multisensory integration: Maintaining the perception of synchrony. Current Biology, 13, R519-R521. Stein, B. E., London, N., Wilkinson, L. K., & Price, D. P. (1996). Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis. Journal of Cognitive Neuroscience, 8, 497-506. Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press. Stone, J. V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., & Porter, N. R. (2001). When is now? Perception of simultaneity. Proceedings of the Royal Society (B), 268, 31-38. Stuss, D. T., Shallice, T., Alexander, M. P., & Picton, T. W. (1995). A multidisciplinary approach to anterior attentional functions. Annals of the New York Academy of Sciences, 769, 191-209. 274

BIBLIOGRAPHY

Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212-215. Taylor K. I., Moss, H. E., Stamatakis, E. A., & Tyler, L. K. (2005). Cross-modal integration and the perirhinal cortex. Journal of Cognitive Neuroscience, 17, Suppl. F299. Theeuwes, J., Godjin, R., & Pratt, J. (2004). A new estimation of the duration of attentional dwell time. Psychonomic Bulletin & Review, 11, 60-64. Thesen, T., Vibell, J., Calvert, G. A. & Osterbauer, R. (2004). Neuroimaging of multisensory processing in vision, audition, touch and olfaction. Cognitive Processing, 5, 84-93. Thomas, G. J. (1941). Experimental study of the influence of vision on sound localization. Journal of Experimental Psychology, 28, 163-177. Titchener, E. B. (1908). Lectures on the elementary psychology of feeling and attention. New York: Macmillan. Tombu, M., & Jolicoeur, P. (2002). All-or-none bottleneck versus capacity sharing accounts of the psychological refractory period phenomenon. Psychological Research, 66, 274-286. Tombu, M., & Jolicoeur, P. (2003). A central capacity sharing model of dual-task performance. Journal of Experimental Psychology: Human Perception & Performance, 29, 3-18. Tse, P. U. (2005). Voluntary attention modulates the brightness of overlapping transparent surfaces. Vision Research, 45, 1095-1098. Turatto, M., Benso, F., Galfano, G., Gamberini, L., & Umiltà, C. (2002). Non-spatial attentional shifts between audition and vision. Journal of Experimental Psychology: Human Perception & Performance, 28, 628-639. 275

BIBLIOGRAPHY

Turatto, M., Galfano, G., Bridgeman, B., & Umiltà, C. (2004). Space-independent modality-driven attentional capture in auditory, tactile and visual systems. Experimental Brain Research, 155, 301-310. Tyler, L. K., & Moss, H. E. (2001). Towards a distributed account of conceptual knowledge. Trends in Cognitive Sciences, 5, 244-252. Uetake, K., & Kudo, Y. (1994). Visual dominance over hearing in feed acquisition procedure of cattle. Applied Animal Behaviour Science, 42, 1-9. Vallar, G. (1993). The anatomical basis of spatial hemineglect in humans. In I. H. Robertson & J. C. Marshall (Eds.), Unilateral neglect: Clinical and experimental studies (pp. 27-59). Hove, UK: Lawrence Erlbaum. Van Damme, S., Crombez, G., Hermans, D., Koster, E. H. W., & Eccleston, C. (2006). The role of extinction and reinstatement in attentional bias to threat: A conditioning approach. Behaviour Research & Therapy, 44, 1555-1563. Van Erp, J. B. F., & Van Veen, H. A. H. C. (2004). Vibrotactile in-vehicle navigation system. Transportation Research Part F, 7, 247-256. Van Wassenhove, V., Grant, K. W., & Poeppel, D. (2001). Temporal window of integration in auditory-visual speech perception. Neuropsychologia, 45, 598607. Vatakis, A., & Spence, C. (2006). Factors modulating the temporal perception of audiovisual speech stimuli. Paper presented at the 7th Annual Meeting of the International Multisensory Research Forum, June 18-21, Dublin, Ireland. Vatakis, A., & Spence, C. (2007a). Investigating the factors that influence the temporal perception of complex audiovisual events. Proceedings of EuroCogSci07, 397-400.

276

BIBLIOGRAPHY

Vatakis, A., & Spence, C. (2007b). Crossmodal binding: Evaluating the ‘unity assumption’ using audiovisual speech stimuli. Perception & Psychophysics, 69, 744-756. Vogel, E. K., & Luck, S. J. (2002). Delayed working memory consolidation during the attentional blink. Psychonomic Bulletin & Review, 9, 739-743. Vogel, E. K., Luck, S. J., & Shapiro, K. L. (1998). Electrophysiological evidence for a postperceptual locus of suppression during the attentional blink. Journal of Experimental Psychology: Human Perception & Performance, 24, 1656-1574. Vroomen, J., & de Gelder, B. (2004). Temporal ventriloquism: Sound modulates the flash-lag effect. Journal of Experimental Psychology: Human Perception & Performance, 30, 513-518. Vroomen, J., & Keetels, M. (2006). The spatial constraint in intersensory pairing: No role in temporal ventriloquism. Journal of Experimental Psychology: Human Perception & Performance, 32, 1063-1071. Ward, R., Duncan, J., & Shapiro, K. (1997). Effects of similarity, difficulty, and nontarget presentation on the time course of visual attention. Perception & Psychophysics, 59, 593-600. Warren, D. H., & Rossano, M. J. (1991). Intermodality relations: Vision and touch. In M. A. Heller & W. Schiff (Eds.), The psychology of touch (pp. 119-137). London: Lawrence Erlbaum Associates. Watanabe, K., & Shimojo, S. (1998). Attentional modulation in perception of visual motion events. Perception, 27, 1041-1054. Watanabe, K., & Shimojo, S. (2001). When sound affects vision: Effects of auditory grouping on visual motion perception. Psychological Science, 12, 109-116.

277

BIBLIOGRAPHY

Watkins, S., Shams, L., Tanaka, S., Haynes, J.-D., & Rees, G. (2006). Sound alters activity in human V1 in association with illusory visual perception. Neuroimage, 31, 1247-1256. Watt, R. J. (1991). Understanding vision. London: Academic Press. Welch, R. B. (1999). Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In G. Ashersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 371-387). Amsterdam: Elsevier Science, B.V. Welch, R. B., DuttonHurt, L. D., & Warren, D. H. (1986). Contributions of audition and vision to temporal rate perception. Perception & Psychophysics, 39, 294300. Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 3, 638-667. Welch, R. B., & Warren, D. H. (1986). Intersensory interactions. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and performance: Vol. 1. Sensory processes and perception (pp. 25-1 - 25-36). New York: Wiley. Welford, A. T. (1952). The ‘psychological refractory period’ and the timing of highspeed performance - A review and a theory. British Journal of Psychology, 43, 2-19. Whipple, G. M., Sanford, E. C., & Colgrove, F. W. (1899). Minor studies from the psychological laboratory of Clark University: On nearly simultaneous clicks and flashes: The time required for recognition: Notes on mental standards of length. American Journal of Psychology, 10, 280-295.

278

BIBLIOGRAPHY

Woodworth, R. S., & Schlosberg, H. (1954). Experimental psychology (Rev. Ed.). New York: Holt, Rinehart & Winston. Yantis, S. (1998). Control of visual attention. In H. Pashler (Ed.), Attention (pp. 223256). London, UK: University College London Press. Yao, L., & Peck, C. K. (1997). Saccadic eye movements to visual and auditory targets. Experimental Brain Research, 115, 25-34. Zampini, M., Shore, D. I., & Spence, C. (2003a). Audiovisual temporal order judgments. Experimental Brain Research, 152, 198-210.
Zampini, M., Shore, D. I., & Spence, C. (2003b). Multisensory temporal order judgments: The role of hemispheric redundancy. International Journal of Psychophysiology, 50, 165-180. Zampini, M., Guest, S., Shore, D. I., & Spence, C. (2005a). Audio-visual simultaneity judgments. Perception & Psychophysics, 67, 531-544. Zampini, M., Shore, D. I., & Spence, C. (2005b). Audiovisual prior entry. Neuroscience Letters, 381, 217-222.

279



Add Comment

To add a comment, you need to login or signup (you can signup without leaving this page)