The Mass General AR/VR RAD Lab: Exploring Virtual and Augmented Reality for Education and Clinical Care
In This Article
- Virtual reality (VR) and augmented reality (AR) technologies have long shown promise for medical training and clinical applications
- Advances in recent years have helped make such applications a reality
- At Massachusetts General Hospital, the recently launched AR/VR RAD Lab is developing and deploying VR and AR technologies for a host of applications
- Training applications include simulation training with "immersive reality" and anatomy education
- Potential clinical applications include presurgical planning and intraprocedural image overlay
Researchers have been exploring the possibilities of using virtual reality (VR) and augmented reality (AR) for more than half a century. It's only in recent years, though, that the technologies have matured to the point where they are ready for widespread implementation in medicine.
Subscribe to the latest updates from Radiology Advances in Motion
Today, researchers at Massachusetts General Hospital, including interventional radiologist Raul N. Uppot, MD, and colleagues, are leading the charge in applying AR and VR for training and pursuing its potential for clinical use—under the banner of the Mass General AR/VR RAD Lab. Sponsored by Radiology Strategic Communications, the lab serves as a research and clinical resource for all of Mass General. At the same time, through collaborations both within the hospital and with outside companies, Dr. Uppot and colleagues in the lab are exploring the integration of AR and VR tools in training and clinical settings.
What Exactly Are Virtual and Augmented Reality?
Broadly defined, virtual reality is an immersive simulated environment that users can explore and even interact with in three dimensions. While the term itself has been in use since the 1930s, when a French avant-garde playwright coined it to describe the illusory nature of the theater, modern understandings of virtual reality can be traced to the 1960s and work by American computer scientist Ivan E. Sutherland. In a 1968 paper, Sutherland and colleagues reported the first head-mounted display that could render new images of a scene accordingly as the viewer moved his or her head, establishing a basic approach to virtual reality still in use today.
While virtual reality environments were originally simply wire-frame model rooms, VR content today can take several different, sophisticated forms, ranging from panoramic stereoscopic videos to computer-aided design (CAD) modeling and digital composition. (Because some of these environments are not, strictly speaking, "virtual" but rather recordings of "actual" reality, they might better be described as immersive environments.)
Augmented reality, as the name suggests, does not put the user in an entirely new, immersive environment, but rather superimposes hologram-like images onto views of the real world. AR technology takes advantage of see-through head-mounted displays, projection-based devices or, increasingly, smartphones and tablets, where the supplementary AR information appears on a device's screen as if it were also in viewing range of its built-in camera.
Immersive Reality for Simulation Training
There is a considerable body of literature demonstrating the pedagogic advantages of three-dimensional immersive learning environments. The greater experiential learning afforded by such environments has been linked with, for example, higher levels of engagement in students and trainees and an improved ability to apply what they have learned in real-world situations. At the same time, 3D virtual learning imparts greater spatial understandings of the material compared to books and other '2D' options.
Today, immersive learning environments are widely used for medical training applications. In the Massachusetts General Hospital Department of Radiology, Dr. Uppot and colleagues have developed and deployed a host of applications that use virtual and augmented reality to help with medical simulation training.
At the most basic level, they have made point-of-view recordings of interventional procedures or acquired 360-degree immersive recordings of the interventional room. The first-person recordings are acquired using two small cameras in a transparent housing mounted on the physician's head; the 360-degree recordings, using seven separate cameras mounted on a tripod and positioned in the center of the room. Specialized software is used to reconstruct images from the seven-camera system.
The immersive recordings are then presented to medical students during training sessions. Using their own smartphones, the students scan a QR code and are transported to the interventional suite, which they explore through plastic stereoscopic viewers they have attached to the phones. As they look around the suite, they are quizzed about equipment and tools they see and learn how they are used for image-guided medical procedures.
Figure 1
Using plastic stereoscopic viewers attached to their smartphones, medical students can explore the hospital's interventional suites in 360 degrees, acclimating themselves even before enter the suites. You can explore the suites much like the students would by clicking and dragging in the video below.
Augmented Reality for Anatomy Training
Another application is the use of augmented reality in anatomy education, where Digital Imaging and Communications in Medicine (DICOM) images can be converted to 3D object files that trainees can engage using head-mounted displays—inspecting images of the heart or inner ear, for example, from any angle while also interacting with one another in real, physical space. Dr. Uppot and colleagues have explored this approach in collaboration with Case Western Reserve Medical School, testing the medical school's HoloAnatomy software suite in small groups of medical students wearing a HoloLens 2 head-mounted mixed-reality display (Microsoft Corp., Redmond, Wash.). Initial feedback from the students suggests AR could offer a promising means of teaching anatomy.
Shown here is video taken by medical students testing the HoloAnatomy software suite. Note that the anatomy images appear as higher quality in person than in the video.
Applications for Clinical Care
The AR/VR RAD Lab at Mass General is also looking to integrate AR and VR technologies into clinical care. For example, through a collaboration with the company Luxsonic (Luxsonic Technologies, Saskatoon, Saskatchewan), Dr. Uppot and colleagues are exploring the potential of virtual reading rooms. Currently, Mass General radiologists interpret images in physical spaces with multiple PACS workstations where they can either work alone or in collaboration with others. Immersive virtual reality offers an opportunity to create similar environments with physicians in different locations, whether in another part of the building or on the other side of the world, interacting as if they were in the same room. Here, using immersive reality headsets, radiologists and referring clinicians can occupy the same virtual space, where they can consult or collaborate with one another while viewing the same medical images.
Other possible applications in the clinic include presurgical planning and intraprocedural image overlay. With respect to the former, the lab is collaborating with DICOM Director (DICOM Director, New Haven, CT) to create interactive 3D holograms of medical images for possible use in presurgical planning. The holograms could enable surgeons to view and interrogate patients' complex cardiac, lung and brain anatomy in three dimensions prior to procedures. The lab is also working with Novarad (Novarad, American Fork, Utah) to develop AR technologies to overlay medical images onto patients for guidance during the procedure itself.
Learn more about Radiology Research at Mass General
Learn more about the Department of Radiology