Game Research Special Interest Group: Multisensory Mixed Reality Games

Understanding behaviour and cognition in dynamic settings

Motivation and Impact

Mixed reality (Virtual Reality, Augmented Reality, a.o.) is a promising technology to better understand (human) behaviour and (social) cognition in more dynamic, interactive, and complex settings. As the real world around us is loaded with multisensory stimuli (vision, audition, haptics), it is of utmost importance to incorporate this in our mixed reality game setups. The aim of this SIG is to foster (longer-term) community building and thus collaborations (e.g. joined projects, joined grant applications (European, national) within and across several scientific disciplines within the Utrecht University (and (preferred) partners) with the ultimate goal to connect researchers from different fields and retain an active lively community even after the official termination of the Focus Area ‘Game Research’. Mixed reality games will pave the way for better, more detailed assessment of 'natural (cognitive) behaviour' from both a fundamental perspective and in clinical applications. With respect to the latter, it might make a huge contribution to, for example, better prediction models of recovery after brain damage and to developing novel treatment paradigms.

For decades, the emphasis has been on visual perception, yet other (multi)sensory domains have also been studied in healthy participants and different clinical populations. Integration of (multi)sensory stimuli is of great importance for efficient behaviour. Adding (multi)sensory stimuli/stimulation, as well as different psychophysiological measurements (e.g. kinematic recordings, eye tracking, EEG, skin conductance, heart rate) to current mixed reality setups will improve our scientific knowledge, skills, infrastructure, and studies, and hence societal impact. 


The aim of this SIG is to bring together different groups of researchers, e.g. from behavioural science (human, animal), informatics, game technology, machine learning, etc. to foster and push research on multisensory mixed reality games in areas such as human behavior understanding and training. We will include our societal partners as much as possible, at different phases in the projects.

Research questions are:

  • Which multisensory outcome measures are relevant for measuring efficient behaviour in complex, mixed reality setups?
  • How to design and model multisensory mixed reality games in fundamental behavioural as well as clinically applied settings?

We will stimulate, facilitate, and participate in smaller high-risk high-gain projects that leverage multisensory stimulation applied to mixed reality to improve a scientific (sub)field. Our method of achieving these aims is to:

  • organize speed presentations showcasing currently running projects applying (multi)sensory stimulation in mixed reality (3x year (e.g. morning));
  • organize meetings on the application of (multi)sensory stimulation in mixed reality to specific fields, or the solution to generic problems across fields, that aim to stimulate joint project proposals (3x year);
  • put together a multisensory mod game environment to kick-start new joint projects;
  • organise a symposium (1x 2 years).

We aim to facilitate and design novel methods for scientific projects and joined grant proposals. Potential grants are NWA, TTW, Synergy, NWO, ERC. We will start with a kick-off project that will be the first showcase of the SIG (e.g. VR simulations with visual, auditory and/or tactile stimulation as multisensory domains, and EEG and/or skin conductance as psychophysical outcome measures).

Team members

  • Tanja Nijboer, Albert Postma, Chris Dijkerman, Anouk Keizer, Faviola Brugger-Dadis (Experimental Psychology, Faculty of Social Sciences)
  • Wolfgang Hurst, Peter Werkhoven, Remco Veltkamp, Egon van den Broek (Information and Computing Sciences, Faculty of Sciences)
  • Joost Raessens (Faculty of Humanities)



  • Virtual Reality and the rehabilitation of cognitive deficits following a stroke