Imagine a chemistry experiment being streamed to thousands of kids on social media. Imagine the kids wearing virtual reality headsets, all focused on the experiment about to begin. They watch a white-coated scientist who carefully pipettes green liquid into a test tube, cuts a piece of a yellow substance and uses tweezers to drop it into the liquid.
Light flashes! Smoke erupts! The kids gasp in awe – but not just because of any old video.
Yes, each kid saw and heard the heat-emitting chemical reaction of the experiment in their VR headset. But they also smelt the pungent odour emitted as a result, felt a warm rush of air blowing through their hair, as well as a slight vibration on their bodies.
Fantasy? Well, this is not as far-fetched as one might think, for kids today could be well on their way towards emulating Wade, the hero of Steven Spielberg’s latest epic Ready Player One, who spends his time in a ubiquitous multi-sensory VR world which has absorbed many of life’s services.
Indeed, this is precisely the remit of the NEWTON project: Networked Labs for Training in Sciences and Technologies for Information and Communication. It’s an EU-wide cooperation involving 14 partners from seven European countries, developing and experimenting with mulsemedia for new technology-based teaching and learning solutions.
Unlike traditional multimedia content, which targets the senses of sight and hearing, mulsemedia – standing for multiple sensorial media – targets three or more human senses. So mulsemedia applications are those that also additionally appeal to at least one of the senses of smell (olfaction), touch (haptics), heat/cold (thermoception) and taste.
Whereas Wade logs in to the videogame paradise OASIS to escape his terrible life, the NEWTON project is using augmented reality and gamification to provide students in the STEM subjects (science, technology, engineering and maths) with engaging and enhancing experience. We are developing user interfaces – ways in which humans interact with systems – that are personalised, multi-modal and adaptive, and which allow for best use of novel e-learning services.
A few decades back, having such digital mulsemedia applications might indeed have been the realm of fantasy, but recently there has been a proliferation of commercial, off-the-shelf devices which can produce such novel user sensations, and can be combined to provide a more engaging learning experience.
For example, coming back to our chemistry experiment, there is no reason why the VR headset worn by the child cannot contain a small tube mounted where it rests on the bridge of the nose, and which can dispense directed, synthesised aromas to the user. One could just as easily imagine a small ventilator fan integrated with the headset near the child’s temple, taking advantage of the heat generated by the device to blow shots of warm air, whilst the vibration felt is produced by a haptic vest vibrating in sync with received audio impulses.
To make matters even more interesting, 360 degree videos can also be used to enhance the user’s sense of immersion, whilst the glasses themselves can contain eye trackers which follow the user’s gaze, allowing them to focus and zoom in on regions of interest through gaze-contingent displays.
Whilst the opportunity to increase the user sense of immersion is undoubtedly a strong driver behind mulsemedia applications, there are others as well.
In the non-digital world, aromatherapy is a well-known and accepted method of therapeutic intervention. However, technology is such nowadays that there is no reason why one could not conceive of a digital spa incorporating aromatherapy at its core.
While mulsemedia applications most certainly have the potential to add a new dimension to experiential-based learning, as in the case of our chemistry experiment, one could also use scents such as rosemary – known to assist memory and cognitive processes – in addition to traditional audiovisual material on e-learning platforms to give users an extra boost in their studies.
Ever wanted to smell a perfume before buying it online, or experience the taste of an exotic fruit without going to the tropics? Again, mulsemedia applications could help out.
It’s often quoted that “Seeing is believing”. Perhaps not so well known is the fact that the complete idiom – as penned by its author, the 17th-century English clergyman Thomas Fuller – is actually “Seeing is believing, but feeling is the truth.”
Whilst I certainly do subscribe to this statement, I do feel that, for learners of the future, the idiom is (at least) three sentences too short.