NA-MIC Project Weeks

Back to Projects List

Collaborative VR breakout session

Key Investigators

• Csaba Pinter (Ebatinca / Pixel Medical)
• Simon Drouin (ETS Montreal)

Project Description

Virtual Reality (VR) has a great potential to facilitate communication between clinicians. For example, VR can be used to collaboratively plan a surgical case by manipulating 3D models derived from preoperative scans. Slicer already includes most of the components required to build prototypes for this kind of applications:

• The Virtual Reality module can display everything from Slicer’s 3D in VR
• The VolumeRendering and PRISM Rendering modules enable advanced-programmable volume rendering
• The OpenIGTLinkIF module and OpenIGTLink protocol enables communication of medical and tracking data between different devices running Slicer The Goal of this discussion is to coordinate future development of the above module to enable more natural collaborative interaction in VR.

Objective

1. Review the current functionality of SlicerVR and PRISM Rendering
2. Discuss planned near future developments in those modules
3. Lay out a vision for the future of interactive VR in Slicer
4. Establish a protocol for Collaborative VR
5. Points for discussion:
   1. Future possibilities for interacting in VR: hand tracking, tool tracking
   2. Need to support different types of tracking in collaborative VR protocols
   3. Should SlicerVR have a fixed set of functionality and interaction paradigm or let users and/or developers choose.

Progress and Next Steps

1. Potential application of SlicerVR: Learn dental anatomy (Sébastien Erckelbout)
2. Short term changes:
   1. Implement spatial references (e.g. a Floor)
   2. Improve object selection
3. Interaction
   1. Is it worth sharing a code base with Slicer Looking glass?
   2. For developers: need for more control over interaction: disable existing interaction and change behavior
   3. Long term: support for more complex VR controllers and hand tracking
   4. Have interaction settings for different scenarios (inside-out vs outside-in visualization)
4. Collaboration
   1. Currently, collaboration is setup using OpenIGTLinkIF and carefully setting up scenes on both ends. Transforms have to be setup for controllers and HMD
   2. We need to facilitate (automate) connection and scene setup for collaboration
   3. It might be necessary to use a server technology to hold common state (Mike Halle suggested FireBase previously used)
   4. A matchmaking and communication solution to explore: Photon

Illustrations

Background and References

SlicerVR

PRISMRendering

PRISMRendering doc