Topics of research

At present, our research is mainly concerned with the question of how information is integrated from multiple sensory modalities into a coherent percept of the world, with a special interest in the question of how visual perception is affected by other sensory modalities. This is a really exciting time to study perception, because we are amidst a paradigm shift -- yes a paradigm shift! For more than a century, perception has been viewed as a modular function with different sensory modalities operating largely as separate and independent modules. (As a result, multisensory integration has been one of the least-studied areas of research in perception.) However, over the course of last few years, accumulating evidence for cross-modal interactions in perception has led to a new surge of interest in this field making it arguably one of the fastest-growing areas of research in perception, and has swiftly overturned the long-standing modular view of perceptual processing. Our research has been among those that have started a shift towards an integrated and interactive paradigm of sensory processing.

We have also recently discovered that crossmodal interactions also play an important role in learning. What is most surprising about this is that multisensory training improves learning even when the task is performed with one modality alone! More specifically, people who get trained with both auditory and visual stimuli perform better (compared to those trained only with visual stimuli) even when sound is no longer available. How could this be? We are currently investigating both the computational and brain mechanisms of this crossmodal facilitation of learning. Auditory-visual training not only accelerates learning but also enhances the magnitude of learning. We think because learning in natural settings almost always occurs in a multisensory setting, the mechanisms of learning have evolved to operate in those conditions and therefore are most effective in multisensory conditions.

We have also recently found that automatic implicit learning of regularities in the environment (statistical learning) can occur simultaneously and independently in visual, and auditory domains as well as across the two modalities. In other words, people can learn three kinds of regularities/associations all at the same time and without any interference among them -- again consistent with the proposition that learning mechanisms have evolved to operate in multisensory conditions, and are most effective (and amazingly so) as such.

Levels of Study

Phenomenology: This is to find out how the different modalities interact at a descriptive level. We investigate the phenomenology of these interactions using behavioral experiments.

Brain Mechanism: This is to find out which brain areas and pathways are involved, in what kind of circuiry (bottom-up, top-down, etc.), and how each area or mechanism contribute to processes of multisensory perception and learning. We have been using event-related brain potentials and functional neuroimaging to investigate these questions. We are also collaborating with neurophysiologists making single-unit recordings in awake behaving monkeys.

Computational Principles: This is to find out what the general theoretical rules governing multisensory perception and learning are. To gain insight into these general principles, one needs to find a model that can account for the behavioral data. We have been using statistical modeling to gain insight into these rules and principles.

Methods of Study

- Altered reality system This is a portable and emersive system that allows the subject to move around, inside or outside of the lab, performing daily tasks while the images get altered in real-time and projected to the head-mounted display that the subject views, in effect altering the subject's "reality". This system can be used to investigate how people adapt to changes in the environment.

- fMRI

- EEG/ERP
  

- Computational Modeling
 

Summary of our some of our recent research

Vision has been traditionally viewed as the dominant modality; one that operates independently of the other modalities. This general view has been changing in the light of accumulating evidence documenting the modulation of visual percepts by signals from other sensory modalities. This shift in mentality, however, has been slow up until recently, since the reports of cross-modal modulation of the visual percept have involved only small, quantitative and gradual changes in the percept. The sound-induced flash illusion, which we recently reported (Nature 2000), on the other hand, provides ground for a decisive change in the traditional view by demonstrating a radical alteration of the visual percept, i.e. an alteration in its phenomenological quality. It also suggests that these modulations are more commonplace than previously thought.

Brain mechanisms underlying multisensory integration

Our ERP and MEG studies suggest that the neural activity associated with the presentation of a visual stimulus is modulated by sound with a very short latency and in areas of the brain thought to process only vision. Our functional Magnetic Resonance Imaging (fMRI) studies confirmed that activity in primary visual cortex (V1) is selectively enhanced in trials in which sound alters the visual perception (i.e., when the illusion occurs) and not otherwise. This is an astonishingly early level of sensory processing, and thus, these results taken together show that even visual processing (which has been thought to be quite self-contained) can be modulated by sound and at the earliest stage of processing in the cortex.

Modeling multi-modal perception

Modeling studies are necessary for gaining insight into the principles underlying cross-modal interactions. We compared human performance in multiple tasks with that an ideal observer that uses Bayesian statistical inference to infer which stimuli correspond to one object and which stimuli correspond to different objects, and how to combine the information from different modalities to obtain the optimal estimate about the world. We found that human performance is remarkably consistent with that of a Bayesian ideal observer. In other words, it seems that evolution has figured out the best computational strategy for accomplishing these perceptual tasks, and found a way of implementing such probabilistic computations in the brain using our neural machinery. Astonishing!

Applications of our research

Our research on learning has important implications for education and rehabilitation. We are currently trying to apply our findings and methods to stroke rehab as well as training protocols in education.

[top↑]