Developing augmented reality and virtual reality in health
This article is from l'Edition n°18
Often confined to the realms of video games or science fiction stories, augmented reality and virtual reality have a huge range of possible applications, particularly in the future of medicine. Researchers at Université Paris-Saclay are considering a number of trials for these technologies.
Augmented reality (AR) offers the opportunity to superimpose virtual elements on a real field of vision through the use of devices (headsets, glasses, etc.). Virtual reality (VR) immerses users in an entirely virtual world, often with the help of an integrated headset. These technologies, both of which are more than 50 years old, have experienced spectacular growth recently. Virtual reality headsets are increasingly being marketed by the entertainment giants (Oculus Quest by Meta, Playstation VR by Sony, etc.) who are aiming to design ever-increasing immersive video games, or even enter the metaverse (an entirely virtual world).
Augmented reality glasses have been just as attractive to the world's largest multinationals (HoloLens by Microsoft or Google Glasses in particular) with varying degrees of success. However, beyond video games and the metaverse, some of the applications for these technologies are currently revolutionising other areas of society.
Augmented reality and virtual reality for use in medicine
Imagine entirely reconstructed operating theatres in the virtual world where student surgeons test their medical skills. After being trained in virtual reality, these same surgeons move to an operating theatre where augmented reality assists them throughout their operation. This is the bold ambition of the BOPA chair (which stands for ‘Bloc opératoire augmenté’ or augmented operating theatre). This chair, which is the result of a partnership between the AP-HP, the Institut Mines-Télécom, Université Paris-Saclay, Inria Saclay and the ‘Conservatoire national des arts et métiers’ (National Conservatory of Arts and Crafts) or Cnam, aims to transform the analysis and learning of surgical procedures by introducing technologies such as augmented reality and virtual reality into medical training and practice.
How to use these two tools to benefit medicine is also studied at the Computer Science, Bioinformatics and Complex Systems Laboratory (IBISC – Univ. Paris-Saclay, Univ. d’Évry). “In the medical education sector, current technologies, including VR, are well placed to provide improved solutions,” explains Samir Otmane, who leads the Augmented Reality & Ambient Robotics Interaction (IRA²) team at IBISC. This team tries to reconstruct training in virtual reality as faithfully as possible in order to practise medicine. In addition, these scientists have already developed bimanual manipulation exercises in virtual reality and a fully-fledged simulator for locoregional anaesthesia. Connected to a computer and a VR headset, an operation on a virtual patient is started with the help of a device featuring fake surgical forceps, for example.
According to Samir Otmane, the main issue preventing the use of VR has long been a lack of accuracy in the reproduction of technical hand gestures. Virtual simulators simply did not offer a high enough degree of realism for them to serve as a suitable context for medical training. Today, as his team’s work shows, medical training using VR is now possible.
The advantages offered by VR in the field of medical training are many. First of all, as the researcher explains, there is a “need for technological tools to improve medical training. Simulation-based training has acquired an important place since the reform in 2018 of the third cycle of medical studies.” Surgical simulation training is now carried out on artificial organs, limbs and dummies. This represents a significant cost, which has the potential to be reduced by incorporating VR into medical simulation training methods.
AR and VR for the rehabilitation of some patients
In addition to medical training, augmented reality and virtual reality are also set to revolutionise the rehabilitation of patients. The IRA² team at IBISC is focusing on people with movement disorders. “The situation is clear as far as rehabilitation is concerned. The number of people is constantly increasing and there’s a lack of human resources to deal with them. It’s very difficult to access care quickly in some regions of France,” laments Samir Otmane. “One of the issues we’ve tried to answer is can we provide new complementary tools which can be used independently by patients and at home?”
In their recent work with the Ellen Poidatz foundation, researchers from the IRA² team have been focusing on walking rehabilitation techniques for children with cerebral palsy. Using an augmented reality headset, they can play a serious game (an activity which combines fun with a serious purpose) which involves walking as quickly as possible behind a small character who encourages them on. The results of the games are then analysed so that patients can make progress in their rehabilitation at their own speed.
Samir Otmane’s team are also developing other serious games to support the rehabilitation of upper limbs in stroke patients. Using AR and VR, it is possible to offer treatment involving a traditional follow-up in hospital and an ‘independent treatment’ which can be used at home. This involves an augmented reality or virtual reality headset, or just a computer equipped with hand movement detectors. The advantages of this combination are numerous. First and foremost, the independent exercises speed up the rehabilitation process. It is important to start this as soon as possible after a stroke. However, motivation is also an important factor. “We’re particularly interested in children. They don’t really respond well to the usual walking rehabilitation techniques. However, they’re keen on play,” points out Samir Otmane.
Within the Complexity, Innovation, Motor and Sports Activities laboratory (CIAMS – Univ. Paris-Saclay, Univ. d’Orléans), this includes animated virtual characters with facial expressions to study empathetic reactions in response to the emotion expressed by the character. In this way, according to Aurore Meugnot, it is possible “to use virtual reality to learn about empathy.” In her recent research, this specialist in cognitive and affective neuroscience has shown that tiny movements “which we often don’t pay much attention to, like blinking or changes in posture” increase the quality of interaction and elicit more emotions from the observer, including a better empathic response. “The challenge is not so much to add realism to the features of a virtual avatar, but rather to improve the empathetic reaction to the avatar,” adds the researcher. In this way, interactions using VR have the advantage of offering patients with impaired empathy, such as children with autism spectrum disorders (ASD), an additional form of therapy. However, there is no real scientific evidence that children find it easier to interact with a virtual character than with a human being. “What we do know is that children find it easier to enter an imaginary world,” points out Aurore Meugnot.
Valentin Bauer, a PhD student working at the Interdisciplinary Laboratory of Digital Sciences (LISN – Univ Paris-Saclay, CNRS, Inria, CentraleSupélec) has become interested in the creation of multi-sensory platforms which use AR and VR. The aim of these new tools is to improve the care of children with ASD. Supported by associate professor Tifanie Bouchara, Valentin Bauer is particularly interested in the contribution audio makes to virtual immersion. “The audio aspect in this new medium of virtual reality is not as well developed as the graphics,” explains Tifanie Bouchara. The two researchers at LISN, who both have a background in engineering, have developed an augmented reality tool capable of interacting in real time with a piece of music directly linked to an immersive graphic representation. This will lead, in the future, to the creation of a multisensory serious game combining auditory and visual experiences.
Publications:
Ricca A. et al. Comparing touch-based and head-tracking navigation techniques in a virtual reality biopsy simulator. Virtual Reality, 25 (1), 191–208, 2021.
Treal T. et al. Natural human postural oscillations enhance the empathic response to a facial pain expression in a virtual character. Scientific Reports, 2020.
Bauer V. et al. Designing an Extended Reality Application to Expand Clinic-Based Sensory Strategies for Autistic Children Requiring Substantial Support : Participation of Practitioners. International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 2021.