This month saw the first ever “virtual reality operation,” conducted as part of research into how the Oculus Rift headset may benefit surgical training. We examine this and other unexpected collaborations between surgery and gaming technology.

surgeon wearing Oculus Rift headsetShare on Pinterest
Dr. Thomas Gregory, whose surgery was filmed for the Oculus Rift.
Image credit: Rémi Rousseau

In March of this year, Facebook acquired the technology start-up Oculus VR, developers of the Kickstarter-funded Oculus Rift virtual reality (VR) headset, for a total of $2 billion.

News of the purchase was met with hostility from the gaming community. The Rift is considered to be the next major evolution in gaming, and the industry was suspicious of what agenda an information-dealing giant like Facebook would have for the technology.

For instance, immediately following the announced takeover, Markus Persson – creator of the award-winning world-building game Minecraft – abruptly cancelled a deal to bring Minecraft to the Rift, tweeting: “I just cancelled that deal. Facebook creeps me out.”

Gaming enthusiasts were unenthusiastic about Facebook CEO Mark Zuckerburg’s plans for the technology they had helped fund, which, it seemed no longer considered games to be the primary objective of the Rift. Instead, Zuckerburg talked of making the Oculus Rift “a new communication platform.”

“We’re going to make Oculus a platform for many other experiences,” the entrepreneur stated. “Imagine enjoying a court-side seat at a game, studying in a classroom of students and teachers all over the world or consulting with a doctor face-to-face – just by putting on goggles in your home.”

While the fear over next-level gaming technology being reduced to a peripheral for making the Facebook experience ever more immersive is not unreasonable, it could be that the fan response to the acquisition was knee-jerk and reactionary.

Perhaps the Rift does have broader applications than just gaming. For instance, could it be used in health care?

Already, groups are experimenting with the technology in a medical context. Earlier this month, scientists, videographers and surgeons collaborated on the first “Oculus Rift operation.”

smartphone with app iconsShare on Pinterest
The Rift looks set to be the first VR headset to be marketed at consumers and has the support of smartphone and desktop platforms. Virtual reality technology has previously been prohibitively costly.

The project was conceived by Rémi Rousseau, a French engineer and VR enthusiast who, via his physician wife, identified the technology’s potential as a training aid for surgeons. Rousseau partnered with the medical research fundraising body MOVEO Foundation to demonstrate, in a practical sense, what role the Oculus Rift might play in this.

Rousseau explains that using VR for training purposes is not new, with virtual reality simulated environments having been used in military training for more than 35 years.

“The advantages conferred by being able to train novice personnel in a low- to no-risk simulated environment have long been appreciated by the medical community,” he blogged, noting however that the “vision of high-quality VR surgery training is quite obvious but the use hasn’t been adopted on a wide scale.”

The reasons for the low take-up of VR in such settings are largely financial. With the support of Apple, Android, Windows and Linux smartphone and desktop platforms, the Rift looks set to be the first VR headset to be marketed at consumers. Virtual reality technology has previously been prohibitively costly and difficult to integrate with existing software and hardware.

“The Rift won’t be a medical device – Facebook probably doesn’t want to deal with [Food and Drug Administration] issues and the medical market is a small niche compared to their usual audiences,” admits Rousseau.

But the Rift, if it’s widely adopted, could be a precious tool for physicians. As [with] the iPhone, they won’t buy it as a medical tool, but as [with] the iPhone, they will use it as a medical tool. They will use it to share cases, surgeries, 3D imagery, and visualize complex data.”

In Rousseau’s experiment, two synchronized tiny GoPro cameras were mounted onto surgeon Dr. Thomas Gregory’s head as he was performing a total hip replacement. The idea was to capture hi-res 3D footage of the operation from the perspective of the surgeon. When viewed back through the headset of the Rift, the viewer has the sensation of being in the surgeon’s body, watching the procedure through their eyes.

“When you’re a surgeon in training, you always have a task to do during the surgery,” explains Rousseau. “It becomes then difficult to see what the main surgeon is doing. Being able to live a surgery in the surgeon’s shoes thanks to the Rift is very useful to replay the surgery in detail, pause, fast forward or backward. It can also be an inspiring way to share and learn new techniques among surgeons.”

You can view the footage in the video below:

Rousseau claims that this is just the beginning, and the next objective for the team is interaction – being able to take part in a full, simulated surgical procedure.

Given that the current iteration of the Rift costs just $350, while radiology screens currently trade for upwards of $15,000, Rousseau suggests that – if the screen resolution of the VR device is improved – it could even be a viable radiological interpretation tool.

He adds:

It’s clear that the technical advances of the Oculus team (in latency reduction for instance), the systematic sharing of their research and the emergence of skilled VR developers among game developers will boost the field of medical VR for the next decades. There will surely be other medical companies standing on the shoulders of Oculus that will provide more specific medical VR hardware for micro-invasive surgery for instance.”

This is not the first time that gaming technology has been investigated as a surgical training tool, however.

In 2012, researchers at the University of Texas Medical Branch (UTMB) at Galveston devised an experiment involving virtual surgical tools. The researchers compared the proficiency of UTMB residents with that of high school- and college-aged gamers at controlling a robot that replicated the physical functions of surgery – and which also measured skills in 32 categories, including hand-eye coordination, pressure on the controls and time.

You can view the test in action in the video below:

The study reported that the high school-aged gamers (who played 2 hours of video games a day) performed better than the college-aged gamers (who played 4 hours of video games a day) and the medical residents (who had less experience of gaming than the other two groups).

However, when the researchers had the groups perform a complicated surgical technique without the robotic aid from the first test, the UTMB residents were the clear leaders in this experiment, which tested a different set of cognitive traits.

Although the study does not suggest that, for surgery, trained medical residents should be moved aside in favor of Call of Duty-obsessed teens, lead author Dr. Sami Kilic – associate professor and director of minimally invasive gynecology in the department of obstetrics and gynecology at UTMB – says that the results have implications for surgical training.

“The inspiration for this study first developed when I saw my son, an avid video game player, take the reins of a robotic surgery simulator at a medical convention,” Dr. Kilic explains. “With no formal training, he was immediately at ease with the technology and the type of movements required to operate the robot.”

Most physicians in practice today never learned robotic surgery in medical school. However, as we see students with enhanced visual-spatial experience and hand-eye coordination that are a result of the technologically savvy world they are immersed in, we should rethink how best to teach this generation.”

A 2013 study from researchers at the University of Rome in Italy, published in PLOS One, also investigated the benefits of a 4-week structured training program using the Nintendo Wii to prepare trainee surgeons for conducting minimally invasive surgery (laparoscopy).

In the study, 42 postgraduate residents were trained to use a validated laparoscopic simulator, and were then randomized into groups that either undertook training with a Nintendo Wii or had no Wii training.

Students in the Wii group played tennis and battle games on the console for 1 hour, 5 days a week, for 4 weeks. The researchers found that the Wii students scored higher than the controls when subsequently tested on the standard surgical simulators.

They attribute this increase in scores to improved hand-eye coordination, honed by the video game training regime, and suggest that the Wii “might be helpful, inexpensive and entertaining part of the training of young laparoscopists, in addition to a standard surgical education based on simulators and the operating room.”

As well as using regular video games to improve hand-eye coordination, some surgeons have also branched out into the games market themselves. Jean Nehme, Andre Chow, Sanjay Purkayastha and Advait Gandhe are a team of UK-based surgeons who developed a “mobile surgical simulator” app, Touch Surgery.

Play the video below to see the app in action:

Vividly detailed, Touch Surgery offers anatomically accurate simulations of 30 different surgical procedures. Each operation is broken down into a series of steps and decision points.

“There is a saying that decision is much more important than incision,” Nehme told British newspaper The Guardian. “Before you cut, you need to be sure of your decision and what comes next. It’s about 75% decision-making and 25% technical skill. The interactive learning process helps you identify risks – at what point should you be aware of this nerve, when should you be aware of this artery.”

The app was created in response to the European working time directive cutting doctors’ hours from around 100 a week to just 40, which the Touch Surgery team says has prevented trainee surgeons from getting the experience they need.

The team says that patients also find the app useful in understanding the procedures they have been referred for, and they actively incorporate feedback from other surgeons.

“A senior consultant told us he often had the same problem with a hip operation,” Nehme said. “A certain vein would always be cut, causing a lot of bleeding, which was very difficult to avoid. But he discovered that dislocating the hip solved this problem entirely. We’re keen to harness the power of different people’s experiences, so their techniques can be shared and discussed.”

To what extent any of these approaches – virtual reality surgery, Super Smash Bros training regimens, or even apps – may transcend their novelty status into viable training aids, though, remains to be seen.

Certainly gaming technology is moving toward a place where its applications reach further outside the niche of video games than ever before, and with an overlap of some skill sets between expert gaming and surgical accuracy, there may be even further ways that these two disciplines could learn from each other.

However, a craze in previous years for mediating training through the online virtual world Second Life seems to have waned more recently as the novelty has worn thin. Could integration with the Oculus Rift bolster interest in this and other surgery-gaming crossovers, or is the Rift-as-surgical aid doomed to follow a similar tapering trajectory?