Medical VR & Segmentation

Improving neurosurgery planning using VR

Affiliated researchers: PD Dr. Tonio BallDr. Lukas Fiederer, Cand. M.Sc. Hisham Alwanni

Fully immersive VR has the potential to improve neurosurgical planning, by offering 3D visualizations of highly detailed anatomical structures with complex shapes, such as blood vessels and tumors. 

A VR research framework for neurosurgery is developed, based on open source tools, for the transformation of patient-specific imaging data into VR 3D representations. The framework extends the conventional planning of surgical interventions by segmenting and modelling patient-specific anatomical structures in 3D and integrating them into an immersive 3D VR environment. Surgeons can use head-mounted displays and wireless controllers to interact with the modelled imaging data in real-time via a user-friendly interface. The interaction capabilities include navigation through the structures, scaling and adjusting the visibility of individual components, applying cross-sectional views and simulating surgical operations with basic mesh manipulations. 

vr_overview.png

Publications:

  • Fiederer L.D.J., Alwanni H., Völker M, Schnell O., Beck J., Ball T., "A Research Framework for Virtual Reality Neurosurgery Based on Open-Source Tools", https://arxiv.org/abs/1908.05188

Medical Image Segmentation, Diagnosis & Interpretability

Affiliated researchers: PD Dr. Tonio BallDr. Lukas Fiederer, Cand. PhD. Robin Schirrmeister, M.Sc. Max Dippel, Cand. M.Sc. Hisham Alwanni

In order to improve the acceptance of deep learning in modern health care, we are trying to develop ways to better understand the reasoning of a neural network. This knowledge can improve the network design, but more importantly give a doctor or his medical staff important meta information to better incorporate automatically generated diagnoses in their workflow.

Brain Tumors

One of our focuses lies on brain tumor segmentation. Given MRI scans in multiple modalities (Flair, T1, T1ce and T2 of the BraTS dataset), we want to segment different parts of a tumor, namely the whole tumor (endema + active tumor + necrotic core), the tumor core (active tumor + necrotic core) and the enhancing tumor (active tumor).

Our goal is, to not only provide highly accurate segmentation maps, but to come up with information, that helps assessing the reasoning and quality of the results, especially when using the network in practice. To that end, we try to measure both importance of parts of the image (Figure 1), as well as uncertainty of the prediction in certain areas (Figure 2).

 Tumor Segmentation Interpretability: Importance

Figure 1: Importance of parts of the input scans for the segmentation prediction.

Tumor Segmentation Interpretability: Uncertainty

Figure 2: Segmentation results for the three target classes (top) and uncertainty of the prediction (bottom).

Vessels

Detailed knowledge about the individual (pathological) layout of cranial blood vessels is crucial for a successful neurosurgical intervention. We use advanced multi-scale spatial filtering techniques to segment blood vessels in as much details as possible.

blood_seg_overview.png

Whole Head Models

Be it using standard clinical MRI or high-end 7 Tesla MRI, our whole head segmentation pipelines try to catch the smallest details as these can have considerable influence on results. The whole head segmentations can be used for as well as for modeling the volume conduction of electromagnetic activity. The latter being a crucial component in the understanding of the spatial aspects of both neuronal activity and brain stimulation, as illustrated in Fiederer et al. 2016a & b.

neuroimage-2016-FEM 

Publications:

  • Fiederer, L.D.J., Lahr J., Vorwerk J., Lucka F., Aertsen A., WoltersC.H., Schulze-Bonhage A., and Ball T., “Electrical Stimulation of the Human Cerebral Cortex by Extracranial Muscle Activity: Effect Quantification With Intracranial EEG and FEM Simulations”, IEEE Transactions on Biomedical Engineering 63, no. 12 (December 2016): 2552–63, doi:10.1109/TBME.2016.2570743.
  • Fiederer, L.D.J., Vorwerk J., Lucka F., Dannhauer M., Yang S., Dümpelmann M., Schulze-Bonhage A., Aertsen A., Speck O., Wolters C.H., and Ball T., “The Role of Blood Vessels in High-Resolution Volume Conductor Head Modeling of EEG”, NeuroImage 128 (March 2016): 193–208, doi:10.1016/j.neuroimage.2015.12.041.

Using EEG to prevent VR-sickness

Affiliated researchers: PD Dr. Tonio Ball, Dipl.Biol. Stephan Hertweck, Cand. MD Desirée Weber, M.Sc. Maryada Bharadwaj

The artificial generation or avoidance of psychophysical effects in virtual environments is an essential principle of most VR systems. Presence, Virtual Body Ownership (VBO), stress and anxiety, for example, are systematically generated and used to facilitate VR applications in areas such as entertainment, therapy, teaching, and training. Undesired psychophysical effects, such as VR-sickness, have to be kept to a minimum.

VR-EEG Foundations

VR systems often are defined by or evaluated for their prominent psychophysical effects like presence, virtual body ownership, and cybersickness. Lately, biometric measures promise to become viable alternatives to the de-facto standard of subjective ratings via questionnaires, since they provide objective and continuous measurements without breaking the exposure. A promising source of information in this context is the EEG which offers a non-invasive window to the brain activity with a millisecond ranged temporal resolution. In collaboration with Prof. Latoschik (HCI Würtzburg) this project investigates the extent to which head-mounted displays (HMDs), influence the quality of EEG measurements. 

VR-Elicited Spectral Artifacts

 

Publications:

  • Hertweck S.*, Weber D.*, Unruh F., Fischbach M., Bharadwaj M., Alwanni H., Latoschik M.E., Ball T., under double-blind review.

 

Personal tools