Publications

In the past decade, virtual reality (VR) and augmented reality (AR) have seen a second wave of activity due to consumer-level availability of devices. In the medical realm, use of VR and AR has been investigated extensively and successfully for training, teaching, rehabilitation, and therapy. However, extended reality (XR) and its intraoperative use in the operating room has yet been underexplored and underdeveloped. Intelligent XR environments that integrate advanced visualization, intuitive interaction, and AI-assisted decision support have the potential to transform clinical care. Responsive, context-aware spaces could enhance medical teams’ capabilities by enabling real-time access to critical information, enhancing situation awareness, and supporting remote collaboration and access to clinical expertise across geographical boundaries—all while preserving the natural flow of procedures. But to realize the potential benefits of XR in the operating room requires a significant and focused effort by an interdisciplinary community. It is the intention of this paper to help to catalyze such an effort. In this paper, we identify pressing issues in clinical care, the potential of XR to address them, and the further research that is needed to realize that potential. We look at a broad range of areas where XR can play an important role, including direct support for the surgeon, support for the surgical team, support for the patient, and remote collaboration. We also consider ways in which XR can be integrated with AI to provide enhanced information and interaction. Discussions are underpinned by considerations of methods to evaluate contributions and progress. This paper is the result of a Dagstuhl Seminar on Extended Reality for the Operating Room (XR4OR), held at the Leibniz Center for Informatics, Schloss Dagstuhl, Germany, in February 2025. It gathered twenty-four researchers working in the areas of virtual reality, augmented reality, medical informatics, human-computer interaction, and surgery.

Using (and developing) haptic devices in virtual reality applications is currently impeded by the lack of flexible haptic rendering software frameworks that support haptic devices with varying numbers of interaction points. The existing open-source framework Chai3D supports primarily haptic devices with single contact points. Game engines are capable of rendering complex scenes, but typically operate at frame rates much lower than 1000 Hz, the update rate required for haptic rendering loops. We present an Unreal Engine-based extension of the UnrealHaptics framework that generalizes the prior single-contact device layer to peripherals with multiple contact points (multi-contact). The work preserves the established high-frequency haptic loop and adapts the framework to Unreal Engine 5 through a generalized device interface and component-based design. The resulting system supports heterogeneous haptic peripherals by exchanging per-contact pose data and six degrees-of-freedom force vectors. We propose a multi-contact performance benchmark and use it to validate our software architecture with two novel haptic peripherals with different numbers of contact points. Also, we propose a novel measurement method for software and hardware latencies, using audio; to quantify the effective visuo-haptic synchronicity of any software-hardware system. We show that our implementation enables fast, efficient, synchronous visuo-haptic rendering, with the haptic loop achieving a 1000 Hz update rate. Also, we demonstrate that our novel measurement method is capable of revealing the bottlenecks (here hardware) of the overall system performance. We also make our framework openly available, aiming at accelerating the development and adoption of haptic interfaces for virtual reality applications.

An increasing number of studies explore gesture control as a sterile and intuitive input modality for medical devices to increase surgeons’ autonomy and reduce errors caused by miscommunication with assisting staff. In this context, gesture feedback is critical, as surgeons must quickly and unambiguously perceive whether an input has been correctly registered. Prior work has compared visual, auditory, and haptic feedback modalities for gesture-based interaction, but often neglects the specific environmental constraints of the operating room (OR), such as high cognitive load and dense acoustic noise from medical equipment. In this paper, we investigate visual, auditory, and haptic gesture feedback under simulated OR conditions. We report on two controlled studies conducted in an OR context with non-surgeons (n = 25) and practicing surgeons (n = 10). Across visual, auditory, and haptic feedback modalities, perceived workload was comparable. However, participants’ qualitative feedback revealed that haptic cues were often perceived as advantageous in terms of unambiguity and reliability in noisy OR environments, although preferences varied substantially across individuals. While performance differences did not reach statistical significance, an observable trend toward higher performance with haptic feedback warrants further investigation and may have practical implications for its use in the OR.

Physics simulations are crucial for domains such as animation and robotics, yet they are limited to deterministic simulations that require precise knowledge of initial conditions. Uncertainty often requires computationally intensive sampling. This paper addresses this challenge by incorporating position and orientation uncertainty into a rigid-body simulation. We formulate and compare four uncertainty propagation approaches: linearization, a surrogate model, persistent sigma points, and local sampling. Our methods build on a penalty-based simulator with a non-uniform sphere hierarchy for geometry approximation and collision detection. We evaluate these methods against Monte Carlo sampling in collision-free and collision scenarios using the Wasserstein distance, and compare their storage requirements. Across our test cases, the proposed methods are about an order of magnitude faster than sampling-based baselines while preserving competitive accuracy. The surrogate model is the fastest, whereas persistent sigma points achieve the best overall accuracy.

Learning minimally invasive endodontic techniques presents unique challenges, requiring precise tooth structure preservation and strong spatial awareness. This study evaluated a clinically realistic virtual reality (VR) simulator, featuring eye-tracking feedback and automated outcome scoring, as an innovative tool to support student learning in minimally invasive endodontics.

Optimal illumination of the surgical site is crucial for successful surgeries. Current lighting systems, however, suffer from significant drawbacks, particularly shadows cast by surgeons and operating room personnel. We introduce an innovative, module-based lighting system that actively prevents shadows using an array of swiveling, ceiling-mounted light modules. The intensity and orientation of these modules are autonomously controlled by novel algorithms utilizing multiple depth sensors mounted above the operating table. This paper presents our complete system, detailing the algorithms for autonomous control and the initial optimization of the light module setup. Unlike prior work that was largely conceptual and based on simulations, this study introduces a real prototype featuring 56 light modules and three depth sensors. We evaluate this prototype through measurements, semi-structured interviews (n=4), and an extensive quantitative user study (n=11). The evaluation focuses on illumination quality, shadow elimination, and suitability for open surgeries compared to conventional OR lights. Our results demonstrate that the novel lighting system and optimization algorithms outperform conventional OR lights for abdominal surgeries, according to both objective measures and subjective ratings by surgeons.

We present a real-time projection mapping system for visualizing information and displaying user interfaces on uneven, deformable surfaces in dynamic environments, where the surface gets partially and dynamically occluded. An important application area is the operating room, where this technology would allow for projection onto the surgical drapes. To achieve precise and adaptive geometric correction in setups with multiple projectors and overlapping, partially occluded projection regions, we adapt a point cloud rendering technique that accurately and efficiently reconstructs surface geometry in the projectors’ image space. This enables an overlap precision of 1.6mm at a projection distance of 2 m, even on uneven surfaces. In addition, we propose two novel GPU-based blur mitigation methods that address blur caused by inevitable inaccuracies of the depth sensors and the overlapping projector images. A user study (𝑛 = 23) shows that our blur mitigation strategies significantly enhance perceived readability, reduce visual artifacts, and lower user workload compared to conventional multi-projector blending. Our system supports projection on arbitrary surfaces, without requiring explicit segmentation and is well-suited to meet the demands of sensitive environments, including those with sterility constraints or limited display access.

Efficient rendering of dynamic point clouds from multiple RGB-D cameras is essential for a wide range of VR/AR applications. In this work, we introduce and leverage two key parameters in a mesh-based rendering approach and conduct a systematic study of their impact on the trade-off between rendering speed and perceptual quality. We show that both parameters enable substantial performance improvements while causing only negligible visual degradation. Across four GPU generations and multiple deployment scenarios, continuous dynamic point clouds from seven Microsoft Azure Kinects can achieve binocular rendering at triple-digit frame rates, even on mid-range GPUs. Our results provide practical guidelines for balancing visual fidelity and efficiency in real-time VR point cloud rendering, demonstrating that mesh-based approaches are a scalable and versatile solution for applications ranging from consumer headsets to large-scale projection systems.

We focus on the relatively underexplored task of packing a set of arbitrary 3D objects—drawn from a predefined distribution—into a single arbitrary 3D container. We simultaneously optimize two potentially conflicting objectives: maximizing the packed volume and maintaining sufficient spacing among objects of the same type to prevent clustering. We present an algorithm to compute solutions to this challenging problem heuristically. Our approach is a flexible two-tier pipeline that computes and refines an initial arrangement. Our results confirm that this approach achieves dense packings across various objects and container shapes.

We present TemPCC, an approach to complete temporal occlusions in large dynamic point clouds. Our method manages a point set over time, integrates new observations into this set, and predicts the motion of occluded points based on the flow of surrounding visible ones. Unlike existing methods, our approach efficiently handles arbitrarily large point sets with linear complexity, does not reconstruct a canonical representation, and considers only local features. Our tests, performed on an Nvidia GeForce RTX 4090, demonstrate that our approach can complete a frame with 30,000 points in under 30 ms, while, in general, being able to handle point sets exceeding 1,000,000 points. This scalability enables the mitigation of temporal occlusions across entire scenes captured by multi-RGB-D camera setups. Our initial results demonstrate that self-occlusions are effectively completed and successfully generalized to unknown scenes despite limited training data.

Indirect vision through a mirror, while bi-manually manipulating both the mirror and another tool is a relatively common way to perform operations in various types of surgery. However, learning such psychomotor skills requires extensive training; they are difficult to teach; and they can be quite costly, for instance, for dentistry schools. In order to study the effectiveness of VR simulators for learning these kinds of skills, we developed a simulator for training dental surgery procedures, which supports tracking of eye gaze and tool trajectories (mirror and drill), as well as automated outcome scoring. We carried out a pre-/post-test study in which 30 fifth-year dental students received six training sessions in the access opening stage of the root canal procedure using the simulator. In addition, six experts performed three trials using the simulator. The outcomes of drilling performed on realistic plastic teeth showed a significant learning effect due to the training sessions. Also, students with larger improvements in the simulator tended to improve more in the real-world tests. Analysis of the tracking data revealed novel relationships between several metrics w.r.t. eye gaze and mirror use, and performance and learning effectiveness: high rates of correct mirror placement during active drilling and high continuity of fixation on the tooth are associated with increased skills and increased learning effectiveness. Larger time allocation for tooth inspections using the mirror, i.e., indirect vision, and frequency of inspection are associated with increased learning effectiveness. Our findings suggest that eye tracking can provide valuable insights into student learning gains of bi-manual psychomotor skills, particularly in indirect vision environments.

Collecting training data for pose estimation methods on images is a time-consuming task and usually involves some kind of manual labeling of the 6D pose of objects. This time could be reduced considerably by using marker-based tracking that would allow for automatic labeling of training images. However, images containing markers may reduce the accuracy of pose estimation due to a bias introduced by the markers. In this paper, we analyze the influence of markers in training images on pose estimation accuracy. We investigate the accuracy of estimated poses for three different cases: i) training on images with markers, ii) removing markers by inpainting, and iii) augmenting the dataset with randomly generated markers to reduce spatial learning of marker features. Our results demonstrate that utilizing marker-based techniques is an effective strategy for collecting large amounts of ground truth data for pose prediction. Moreover, our findings suggest that the usage of inpainting techniques do not reduce prediction accuracy. Additionally, we investigate the effect of inaccuracies of labeling in training data on prediction accuracy. We show that the precise ground truth data obtained through marker tracking proves to be superior compared to markerless datasets if labeling errors of 6D ground truth exist.

Traditional techniques for rendering continuous surfaces from dynamic, noisy point clouds using multi-camera setups often suffer from disruptive artifacts in overlapping areas, similar to z-fighting. We introduce BlendPCR, an advanced rendering technique that effectively addresses these artifacts through a dual approach of point cloud processing and screen space blending. Additionally, we present a UV coordinate encoding scheme to enable high-resolution texture mapping via standard camera SDKs. We demonstrate that our approach offers superior visual rendering quality over traditional splat and mesh-based methods and exhibits no artifacts in those overlapping areas, which still occur in leading-edge NeRF and Gaussian Splat based approaches like Pointersect and P2ENet. In practical tests with seven Microsoft Azure Kinects, processing, including uploading the point clouds to GPU, requires only 13.8 ms (when using one color per point) or 29.2 ms (using high-resolution color textures), and rendering at a resolution of 3580 x 2066 takes just 3.2 ms, proving its suitability for real-time VR applications.

Physics simulations are crucial for domains like animation and robotics, yet they are limited to deterministic simulations with precise knowledge of initial conditions. We introduce a surrogate model for simulating rigid bodies with positional uncertainty (Gaussian) and use a non-uniform sphere hierarchy for object approximation. Our model outperforms traditional sampling-based methods by several orders of magnitude in efficiency while achieving similar outcomes.

Common techniques for anatomy education in medicine include lectures and cadaver dissection, as well as the use of replicas. However, recent advances in virtual reality (VR) technology have led to the development of specialized VR tools for teaching, training, and other purposes. The use of VR technology has the potential to greatly enhance the learning experience for students. These tools offer highly interactive and engaging learning environments that allow students to inspect and interact with virtual 3D anatomical structures repeatedly, intuitively, and immersively. Additionally, multi-user VR environments can facilitate collaborative learning, which has the potential to enhance the learning experience even further. However, the effectiveness of collaborative learning in VR has not been adequately explored. Therefore, we conducted two user studies, each with n1,2 = 33 participants, to evaluate the effectiveness of virtual collaboration in the context of anatomy learning, and compared it to individual learning. For our two studies, we developed a multi-user VR anatomy learning application using UE4. Our results demonstrate that our VR Anatomy Atlas offers an engaging and effective learning experience for anatomy, both individually and collaboratively. However, we did not find any significant advantages of collaborative learning in terms of learning effectiveness or motivation, despite the multi-user group spending more time in the learning environment. In fact, motivation tended to be slightly lower. Although the usability was rather high for the single-user condition, it tended to be lower for the multi-user group in one of the two studies, which may have had a slightly negative effect. However, in the second study, the usability scores were similarly high for both groups. The absence of advantages for collaborative learning may be due to the more complex environment and higher cognitive load. In consequence, more research into collaborative VR learning is needed to determine the relevant factors promoting collaborative learning in VR and the settings in which individual or collaborative learning in VR is more effective, respectively.

Point clouds play an important role in robotics, autonomous driving, and telepresence applications with typical tasks such as SLAM and scene/avatar reconstruction. However, noisy sensor data, huge data loads, and inhomogeneous densities make efficient processing and accurate representation challenging, especially for real-time and streaming-based applications. We present a novel approach for compact point cloud representation and real-time streaming using a temporal hierarchical GMM-based generative model. Our level-based construction scheme allows us to dynamically adjust the maximum LOD and progressively transmit and render more detailed levels. We minimize the construction cost by exploiting the temporal coherence between consecutive frames. Combined with our highly parallelized and optimized CUDA implementation, we achieve real-time speeds with high-fidelity reconstructions. Our results show that we achieve significantly higher compression factors than previous work with similar accuracy, and that the temporal approach saves 20-36% construction time in our test scene.

Good illumination of the surgical site is crucial for the success of a surgery—yet current, typical surgical lighting systems have significant shortcomings, e.g. with regard to shadowing and ease of handling. To address these shortcomings, new lighting systems for operating rooms have recently been developed, consisting of a variety of swiveling light modules that are mounted on the ceiling and controlled automatically. For such a new type of lighting system, we present a new optimization pipeline that maintains the brightness at the surgical site as constant as possible over time and minimizes shadows by using depth sensors. Furthermore, by performing simulations on point cloud recordings of nine real abdominal surgeries, we demonstrate that our optimization pipeline is capable of effectively preventing shadows cast by bodies and heads of the OR personnel.

In medical education, anatomy is typically taught through lectures, cadaver dissection, and using replicas. Advances in VR technology facilitated the development of specialized VR tools for teaching, training, and other tasks. They can provide highly interactive and engaging learning environments where students can immersively and repeatedly inspect and interact with virtual 3D anatomical structures. Moreover, multi-user VR environments can be employed for collaborative learning, which may enhance the learning experience. Concrete applications are still rare, though, and the effect of collaborative learning in VR has not been adequately explored yet. Therefore, we conducted a user study with n= 33 participants to evaluate the effectiveness of virtual collaboration on the example of anatomy learning (and compared it to individual learning). For our study, we developed an UE4-based multi-user VR anatomy learning application. Our results show that our VR Anatomy Atlas provides an engaging learning experience and is very effective for anatomy learning, individually as well as collaboratively. However, interestingly, we could not find significant advantages for collaborative learning regarding learning effectiveness or motivation, even though the multi-user group spent more time in the learning environment. Although rather high for the single-user condition, the usability tended to be lower for the multi-user group. This may be due to the more complex environment and a higher cognitive load. Thus, more research in collaborative VR for anatomy education is needed to investigate, if and how it can be employed more effectively.

Multi-user VR applications have great potential to foster remote collaboration and improve or replace classical training and education. An important aspect of such applications is how participants move through the virtual environments. One of the most popular VR locomotion methods is the standard teleportation metaphor, as it is quick, easy to use and implement, and safe regarding cybersickness. However, it can be confusing to the other, observing, participants in a multi-user session and, therefore, reduce their presence. The reason for this is the discontinuity of the process, and, therefore, the lack of motion cues. As of yet, the question of how this teleport metaphor could be suitably visualized for observers has not received very much attention. Therefore, we implemented several continuous and discontinuous 3D visualizations for the teleport metaphor and conducted a user study for evaluation. Specifically, we investigated them regarding confusion, spatial awareness, and spatial and social presence. Regarding presence, we did find significant advantages for one of the visualizations. Moreover, some visualizations significantly reduced confusion. Furthermore, multiple continuous visualizations ranked significantly higher regarding spatial awareness than the discontinuous ones. This finding is also backed up by the users' tracking data we collected during the experiments. Lastly, the classic teleport metaphor was perceived as less clear and rather unpopular compared with our visualizations.

We present a new method to model the mass of celestial bodies based on adaptive polydisperse sphere packings. Using poly- disperse spheres in the mascon model has shown to deliver a very good approximation of the mass distribution of celestial bodies while allowing fast computations of the gravitational field. However, small voids between the spheres reduce the accuracy especially close to the surface. Hence, the idea of our adaptive sphere packing is to place more spheres close to the surface instead of filling negligible small gaps deeper inside the body. Although this reduces the packing density, we achieve greater accuracy close to the surface. For the adaptive sphere packing, we propose a mass assignment algorithm that uniformly samples the volume of the body. Additionally, we present a method to further optimize the mass distribution of the spheres based on least squares optimization. The sphere packing and the gravitational acceleration remain computable entirely on the GPU (Graphics Processing Unit).

Bistatic radar (BSR) is a well-established technology to probe surfaces of planets and also small bodies like asteroids and comets. The radio subsystem onboard the spacecraft serves as the transmitter and the ground station on Earth as the receiver of the radio signal in the bistatic radar configuration. A part of the reflected signal is scattered towards the receiver which records both the right-hand circular polarized (RHCP) and left-hand circular polarized (LHCP) echo components. From the measurement of those, geophysical properties like surface roughness and dielectric constant can be derived. Such observations aim at extracting the radar reflectivity coefficient of the surface, which is also called the radar-cross section. This coefficient depends on the physical properties of the surface. We developed a bistatic radar simulation tool that utilizes hardware acceleration and massively-parallel programming paradigms available on modern GPUs. It is based on the Shooting and Bouncing Rays (SBR) method (sometimes also called Ray-Launching Geometrical Optics), which we have adapted for the GPU and implemented using hardware- accelerated raytracing. This provides high-performance estimation of the scattering of electromagnetic waves from surfaces, which is highly desirable since surfaces can become very large relative to the surface features that need to be resolved by the simulation method. Our method can, for example, deal with the asteroids 1 Ceres and 4 Vesta, which have mean diameters of around 974 km and 529 km, resp., which are very large surfaces relative to the sizes of the surface features. But even smaller objects can require a large number of rays for sampling the surface with a density large enough for accurate results. In this paper, we present our new, very efficient simulation method, its application to several examples with various shapes and surface properties, and examine limits of the detectability of water ice on small bodies.

We propose a novel method to avoid stereo window violations at screen borders. These occur for objects in front of the zero parallax plane, which appear in front of the (physical) screen, and that are clipped for one eye while still being visible for the other eye. This contradicts other stereo cues, particularly disparity, potentially resulting in eye strain and simulator sickness. In interactive and dynamic virtual environments, where the user controls the camera, e.g., via head tracking, it is impossible to avoid stereo window violations completely. We propose \textsl{Dynparity}, a novel rendering method to eliminate the conflict between clipping and negative disparity, by introducing a non-uniform stereoscopic projection. For each vertex in front of the zero parallax plane, we compute the stereoscopic projection such that the parallax approaches zero towards the edge of the screen. Our approach works entirely on the GPU in real-time and can be easily included in modern game engines. We conducted a user study comparing our method to the standard stereo projection on a large-screen stereo wall with head tracking. Our results show significantly reduced simulator sickness when using Dynparity compared to the standard stereo rendering.

Registration is an essential prerequisite for many applications when a multiple-camera setup is used. Due to the noise in depth images, registration procedures for depth sensors frequently rely on the detection of a target object in color or infrared images. However, this prohibits use cases where color and infrared images are not available or where there is no mapping between the pixels of different image types, e.g., due to separate sensors or different projections. We present our novel registration method that requires only the point cloud resulting from the depth image of each camera. For feature detection, we propose a combination of a custom-designed 3D registration target and an algorithm that is able to reliably detect that target and its features in noisy point clouds. Our evaluation indicates that our lattice detection is very robust (with a precision of more than 0.99) and very fast (on average about 20 ms with a single core). We have also compared our registration method with known methods: Our registration method achieves an accuracy of 1.6 mm at a distance of 2 m using only the noisy depth image, while the most accurate registration method achieves an accuracy of 0.7 mm requiring both the infrared and depth image.

Perception in complex environments especially dynamic and human-centered ones goes beyond classical tasks such as classification usually known as the what- and where-object-questions from sensor data, and poses at least three challenges that are missed by most and not properly addressed by some actual robot perception systems. Note that sensors are extrinsically (e.g., clutter, embodiedness-due noise, delayed processing) and intrinsically (e.g., depth of transparent objects) very limited, resulting in a lack of or high-entropy data, that can only be difficultly compressed during learning, difficultly explained or intensively processed during interpretation. (a) Therefore, the perception system should rather reason about the causes that produce such effects (how/why-happen-questions). (b) It should reason about the consequences (effects) of agent-object and object-object interactions in order to anticipate (what-happen-questions) the (e.g., undesired) world state and then enable successful action on time. (c) Finally, it should explain its outputs for safety (meta why/how-happen-questions). This paper introduces a novel white-box and causal generative model of robot perception (NaivPhys4RP) that emulates human perception by capturing the Big Five aspects (FPCIU) of human commonsense, recently established, that invisibly (dark) drive our observational data and allow us to overcome the above problems. However, NaivPhys4RP particularly focuses on the aspect of physics, which ultimately and constructively determines the world state.

In recent years, a trend has existed for an open benchmark aiming for reproducible and comparable benchmarking results. The best reproducibility can be achieved when performing the benchmarks in the same hard- and software environment. This can be offered as a web service. One challenge of such a web service is the integration of new algorithms into the existing benchmarking tool due to security concerns. In this paper, we present a framework that allows the safe execution of user-uploaded algorithms in such a benchmark-as-a-service web tool. To guarantee security as well as reproducibility and comparability of the service, we extend an existing system architecture to allow the execution of user-uploaded algorithms in a virtualization environment. Our results show that although the results from the virtualization environment are slightly slower by around 3.7% to 4.7% compared with the native environment, the results are consistent across all scenarios with different algorithms, object shapes, and object complexity. Moreover, we have automated the entire process from turning on/off a virtual machine, starting benchmark with intended parameters to communicating with the backend server when the benchmark has finished. Our implementation is based on Microsoft Hyper-V that allows us to benchmark algorithms that use Single Instruction, Multiple Data (SIMD) instruction sets as well as access to the Graphics Processing Unit (GPU).

Current and future small body missions, such as the ESA Hera mission or the JAXA MMX mission demand good knowledge of the gravitational field of the targeted celestial bodies. This is not only motivated to ensure the precise spacecraft operations around the body, but likewise important for landing manoeuvres, surface (rover) operations, and science, including surface gravimetry. To model the gravitation of irregularly-shaped, non-spherical bodies, different methods exist. Previous work performed a comparison between three different methods, considering a homogeneous density distribution inside the body. In this work, the comparison is continued, by introducing a first inhomogeneity inside the body. For this, the same three methods, being the polyhedral method and two different mascon methods are compared.

We present a novel approach of redirected walking (RDW) based on step feedback sounds to redirect users in virtual reality. The main idea is to achieve path manipulation by changing step noises to deviate the users, who still believe that they are walking a straight line. Our approach can be combined with traditional visual approaches for RDW based on eye-blinking. Moreover, we have conducted a user study in a large area (10×20m) using a within-subject design. We achieved a translational redirection of 1.7m in average with pure audio feedback. Moreover, our results show that visual methods can amplify the deviation of our new auditory approach by 80cm in average at the distance of 20 m.

We present a study investigating the question whether and how people’s intention to change their environmental behavior depends on the degrees of immersion and freedom of navigation when they experience a deteriorating virtual coral reef. We built the virtual reef on top of a biologically sound model of the ecology of coral reefs, which allowed us to simulate the realistic decay of reefs under adverse environmental factors. During their experience, participants witnessed those changes while they also explored the virtual environment. In a two-factorial experiment (N = 224), we investigated the effects of different degrees of immersion and different levels of navigation freedom on emotions, the feeling of presence, and participants’ intention to change their environmental behavior. The results of our analyses show that immersion and navigation have a significant effect on the participants’ emotions of sadness and the feeling of helplessness. In addition, we found a significant effect, mediated by the participants’ emotions, on the intention to change their behavior. The most striking result is, perhaps, that the highest level of immersion combined with the highest level of navigation did not lead to the highest intentions to change behavior. Overall, our results show that it is possible to raise awareness of environmental threats using virtual reality; it also seems possible to change people’s behavior regarding these threats. However, it seems that the VR experience must be carefully designed to achieve these effects: a simple combination of all affordances offered by VR technology might potentially decrease the desired effects.

In this study, we present and compare three different methods to model the gravitational field of small bodies and apply them to three test cases that we describe in detail. Our first method is based on the polyhedral method that pro- vides a closed-form analytical solution of the gravity field for (assumed) homogeneous density. The idea behind the second method is to represent the small body’s mass by a polydisperse sphere packing. This allows us an easy and efficient computation through parallelization on the GPU (Graphics Processing Unit). The third method models the internal mass distribution of the body as a set of solid elements in spherical coordinates. The body is divided into longitudes and latitudes and the radius is divided into subsections. The used size of the volume elements is chosen to ensure high accuracy in representing the shape of the body. All three methods are also applicable on the surface of the body, making it interesting in the context of surface gravimetry. We evaluate the three methods using two ideal shapes (sphere and cube) and one real shape model (Martian moon Phobos). We compare the gravitational acceleration at their surface and measure the relative error of the models concerning the analyt- ical solutions. We also look at the computational cost of each method. Our proposed methods indicate that each of them is suitable for modeling asteroids with different characteristics. We provide reliable gravitation data for purposes such as space- craft orbit analysis and evaluation of the small body’s surface domain.

The complex shape of asteroids and comets is a critical parameter in many scientific and operational studies. From the global irregular shape down to the local surface details, these topographies reflect the formation and evolutionary processes that remould the celestial body. Furthermore, these processes control how the surface will continue to evolve: from mass wasting on high slopes to spin-up due to anisotropic re-emission of thermal radiation. In addition, for space missions, the irregular coarse shape and complex landscape are a hazard to navigation, which must be accounted for in the planning phase.

Virtual testbeds (VTBs) are essential for researchers and engineers during the planning, decision making, and testing phases of space missions because they are much faster and cost-effective than physical models or tests. Moreover, they allow to simulate the target conditions that are not available on earth for real-world tests, and it is possible to change or adjust mission parameters or target conditions on-the-fly. However, such highly specialized and flexible tools are often only available as desktop tools with limited visual feedback and a lack of usability. On the other hand, VR is predestinated for easy, natural interac-tion even in complex decision making and training scenarios, while simultaneously offering high fidelity visual feedback and immersion. We present a novel tool that combines the flexibility of virtual testbeds with an easy-to-use VR interface. To do so, we have extended a VTB for planetary exploration missions, the VaMEx-VTB (Valles Marineris Exploration-VTB), to support sophisticated virtual reality (VR) interactions. The VTB is based on the modern game engine "Unreal Engine 4", which qualifies it for state-of-the-art rendering. Additionally, our system supports a wide variety of different hardware devices, including head- mounted displays (HMDs) and large projection powerwalls with different tracking and input methods. Our VR-VTB enables the users to investigate simulated sensor output and other mission parameters like lines-of-sight or ground formations for a swarm of different spacecraft, including autonomous ground vehicles, flying drones, a humanoid robot, and supporting orbiters. Moreover, the users can directly interact with the virtual environment to distract the swarm units or change environment parameters, like adding boulders or invoking sand storms. Until now, we have used our system for three different scenarios: a swarm-based exploration of the Valles Marineris on planet Mars, a test scenario of the same swarm units on the Canary Islands, and the autonomous building of a moon base. An expert review shows the general usability of our VR-VTB.

We present a novel selection technique for VR called LenSelect. The main idea is to decrease the Index of Difficulty (ID) according to Fitts’ Law by dynamically increasing the size of the potentially selectable objects. This facilitates the selection process especially in cases of small, distant or partly occluded objects, but also for moving targets. In order to evaluate our method, we have defined a set of test scenarios that covers a broad range of use cases, in contrast to often used simpler scenes. Our test scenarios include practically relevant scenarios with realistic objects but also synthetic scenes, all of which are available for download. We have evaluated our method in a user study and compared the results to two state-of-the-art selection techniques and the standard ray-based selection. Our results show that LenSelect performs similar to the fastest method, which is ray-based selection, while significantly reducing the error rate by 44%.

The precise registration between multiple depth sensors is a crucial prerequisite for many applications. Previous techniques frequently rely on RGB or IR images and checkerboard targets for feature detection. However, this prohibits the usage for use-cases where neither is available or where IR and depth images have different projections. Therefore, we present a novel registration approach that uses depth data exclusively for feature detection, making it more universally applicable while still achieving robust and precise results. We propose a combination of a custom 3D registration target — a lattice with regularly-spaced holes — and a feature detection algorithm that is able to reliably extract the lattice and its features from noisy depth images. In addition, we have integrated the registration procedure to a publicly available Unreal Engine 4 plugin that allows multiple point clouds captured by several depth cameras to be registered in a virtual environment. Despite the rather noisy depth images, we are able to quickly obtain a robust registration that yields an average deviation of 3.8 mm to 4.4 mm in our test scenarios.

The precise registration between multiple depth cameras is a crucial prerequisite for many applications. Previous techniques frequently rely on RGB or IR images and checkerboard targets for feature detection, partly due to the depth data being inherently noisy. This limitation prohibits the usage for use-cases where neither is available. We present a novel registration approach that solely uses depth data for feature detection, making it more universally applicable while still achieving robust and precise results. We propose a combination of a custom 3D registration target - a lattice with regularly-spaced holes - and a feature detection algorithm that is able to reliably extract the lattice and its features from noisy depth images.

UnrealHaptics is a plugin-architecture that enables advanced virtual reality (VR) interactions, such as haptics or grasping in modern game engines. The core is a combination of a state-of-the-art collision detection library with support for very fast and stable force and torque computations and a general device plugin for communication with different input/output hardware devices, such as haptic devices or Cybergloves. Our modular and lightweight architecture makes it easy for other researchers to adapt our plugins to their requirements. We prove the versatility of our plugin architecture by providing two use cases implemented in the Unreal Engine 4 (UE4). In the first use case, we have tested our plugin with a haptic device in different test scenes. For the second use case, we show a virtual hand grasping an object with precise collision detection and handling multiple contacts. We have evaluated the performance in our use cases. The results show that our plugin easily meets the requirements of stable force rendering at 1 kHz for haptic rendering even in highly non-convex scenes, and it can handle the complex contact scenarios of virtual grasping.

Total hip arthroplasty (THA) is a highly successful surgical procedure, but complications remain, including aseptic loosening, early dislocation and misalignment. These may partly be related to lacking training opportunities for novices or those performing THA less frequently. A standardized training setting with realistic haptic feedback for THA does not exist to date. Virtual Reality (VR) may help establish THA training scenarios under standardized settings, morphology and material properties. This work summarizes the development and acquisition of mechanical properties on hip reaming, resulting in a tissue-based material model of the acetabulum for force feedback VR hip reaming simulators. With the given forces and torques occurring during the reaming, Cubic Hermite Spline interpolation seemed the most suitable approach to represent the nonlinear force-displacement behavior of the acetabular tissues over Cubic Splines. Further, Cubic Hermite Splines allowed for a rapid force feedback computation below the 1 ms hallmark. The Cubic Hermite Spline material model was implemented using a three-dimensional-sphere packing model. The resulting forces were delivered via a human-machine-interaction certified KUKA iiwa robotic arm used as a force feedback device. Consequently, this novel approach presents a concept to obtain mechanical data from high-force surgical interventions as baseline data for material models and biomechanical considerations; this will allow THA surgeons to train with a variety of machining hardness levels of acetabula for haptic VR acetabulum reaming.

Advances in computer technology and increasing usage of computer graphics in a broad field of applications lead to rapidlyrising demands regarding size and detail of virtual landscapes. Manually creating huge, realistic looking terrains and populatingthem densely with assets is an expensive and laborious task. In consequence, (semi-)automatic procedural terrain generationis a popular method to reduce the amount of manual work. However, such methods are usually highly specialized for certainterrain types and especially the procedural generation of landscapes composed of different biomes is a scarcely explored topic.We present a novel system, called AutoBiomes, which is capable of efficiently creating vast terrains with plausible biomedistributions and therefore different spatial characteristics. The main idea is to combine several synthetic procedural terraingeneration techniques with digital elevation models (DEMs) and a simplified climate simulation. Moreover, we include aneasy-to-use asset placement component which creates complex multi-object distributions. Our system relies on a pipelineapproach with a major focus on usability. Our results show that our system allows the fast creation of realistic looking terrains.

Hand tracking is essential in many applications reaching from the creation of CGI movies to medical applications and even real-time, natural, physically-based grasping in VR. Optical marker-based tracking is often the method of choice because of its high accuracy, the support for large workspaces, good performance, and there is no wiring of the user required. However, the tracking algorithms may fail in case of hand poses where some of the markers are occluded. These cases require a subsequent reassignment of labels to reappearing markers. Currently, convolutional neural networks (CNN) show promising results for this re-labeling because they are relatively stable and real-time capable. In this paper, we present several methods to improve the accuracy of label predictions using CNNs. The main idea is to improve the input to the CNNs, which is derived from the output of the optical tracking system. To do so, we propose a method based on principal component analysis, a projection method that is perpendicular to the palm, and a multi-image approach. Our results show that our methods provide better label predictions than current state-of-the-art algorithms, and they can be even extended to other tracking applications.

In clinical practice, medical imaging technologies, like computed tomography, have become an important and routinely used technique for diagnosis. Advanced 3D visualization techniques of this data, e.g. by using volume rendering, provide doctors a better spatial understanding for reviewing complex anatomy. There already exist sophisticated programs for the visualization of medical imaging data, however, they are usually limited to exactly this topic and can be hardly extended to new functionality; for instance, multi-user support, especially when considering immersive VR interfaces like tracked HMDs and natural user interfaces, can provide the doctors an easier, more immersive access to the information and support collaborative discussions with remote colleagues. We present an easy-to-use and expandable system for volumetric medical image visualization with support for multi-user VR interactions. The main idea is to combine a state-of-the-art open-source game engine, the Unreal Engine 4, with a new volume renderer. The underlying game engine basis guarantees the extensibility and allows for easy adaption of our system to new hardware and software developments. In our example application, remote users can meet in a shared virtual environment and view, manipulate and discuss the volume-rendered data in real-time. Our new volume renderer for the Unreal Engine is capable of real-time performance, as well as, high-quality visualization.

We present a server-based benchmark that enables a fair analysis of different collision detection & proximity query algorithms. A simple yet interactive web interface allows both expert and non-expert users to easily evaluate different collision detection algorithms’ performance in standardized or optionally user-definable scenarios and identify possible bottlenecks. In contrast to typically used simple charts or histograms to show the results, we additionally propose a heatmap visualization directly on the benchmarked objects that allows the identification of critical regions on a sub-object level. An anonymous login system, in combination with a server-side scheduling algorithm, guarantees security as well as the reproducibility and comparability of the results. This makes our benchmark useful for end-users who want to choose the optimal collision detection method or optimize their objects with respect to collision detection but also for researchers who want to compare their new algorithms with existing solutions.

We present a novel noise model to procedurally generate volumetric terrain on implicit surfaces. The main idea is to combine a novel Locally Controlled 3D Spot noise (LCSN) for authoring the macro structures and 3D Gabor noise to add micro details. More specifically, a spatially-defined kernel formulation in combination with an impulse distribution enables the LCSN to generate arbitrary size craters and boulders, while the Gabor noise generates stochastic Gaussian details. The corresponding metaball positions in the underlying implicit surface preserve locality to avoid the globality of traditional procedural noise textures, which yields an essential feature that is often missing in procedural texture based terrain generators. Furthermore, different noise-based primitives are integrated through operators, i.e. blending, replacing, or warping into the complex volumetric terrain. The result is a completely implicit representation and, as such, has the advantage of compactness as well as flexible user control. We applied our method to generating high quality asteroid meshes with fine surface details.

We present a novel haptic rendering method to simulate material removal in medical simulations at haptic rates. The core of our method is a new massively-parallel continuous collision detection algorithm in combination with a stable and flexible 6-DOF collision response scheme that combines penalty- and constraint-based force computation. Moreover, a volumetric representation allows us to derive a realistic local material model from experimental human cadaveric data, as well as support real-time continuous material removal. We have applied our algorithm to a hip replacement simulator and two dentistry-related simulations for root-canal opening and caries removal. The results show realistic continuous forces and torques at haptic rates.

We present a novel data structure for SIMD optimized simultaneous bounding volume hierarchy (BVH) traversals like they appear for instance in collision detection tasks. In contrast to all previous approaches, we consider both the traversal algorithm and the construction of the BVH. The main idea is to increase the branching factor of the BVH according to the available SIMD registers and parallelize the simultaneous BVH traversal using SIMD operations. This requires a novel BVH construction method because traditional BVHs for collision detection usually are simple binary trees. To do that, we present a new BVH construction method based on a clustering algorithm, Batch Neural Gas, that is able to build efficient n-ary tree structures along with SIMD optimized simultaneous BVH traversal. Our results show that our new data structure outperforms binary trees significantly.

We present an overview of the Valles Marineris Explorer (VaMEx) initiative, a DLR-funded project line for the development of required key technologies to enable a future swarm exploration of the Valles Marineris on Mars. The Valles Marineris is a wide canyon range, near the Martian equator. The so far still fictive VaMEx mission scenario compromises a swarm of different robots, including rovers, flying drones and a hominid robot. Here, we present VaMEx-VTB , a virtual testbed (VTB) with a digitalized map of the large and frag- mented terrain of the Valles Marineris. The VaMEx-VTB allows an adjustable validation as well as verification of the complex mission design in virtual reality, due to its modular design. It shall also be used in preparation of field tests in the near future for validation of each swarm element’s ability for interactive swarm cooperation and collaboration.

We present a novel haptic rendering approach to simulate material removal in medical simulations at haptic rates. The core of our method is a new massively-parallel continuous collision detection algorithm in combination with a stable and flexible 6-DOF collision response scheme that combines penalty-based and constraint-based force computation.

Chapter Abstract: In this chapter, we present two methods for collision detection in virtual environments. The first method relies on a data structure called the Inner Sphere Tree (IST). ISTs are suitable for rigid objects and they are the first data structure that is able to compute the penetration volume between a pair of colliding objects at haptic rendering rates. This new contact information guarantees physically-plausible and continuous forces and torques for the collision responses that are essential for stable physically-based simulations and haptic rendering. ISTs do rely on a bounding volume hierarchy that requires a time-consuming pre-processing that becomes invalid in case of deformations. Consequently, for deformable objects, we propose another algorithm (we call it kDet) that does not need any pre-processing. kDet works completely on the GPU and has a constant running time for practically all relevant objects.

About the Book:

This book takes the practicality of other "Gems" series such as "Graphics Gems" and "Game Programming Gems" and provide a quick reference for novice and expert programmers alike to swiftly track down a solution to a task needed for their VR project. Reading the book from cover to cover is not the expected use case, but being familiar with the territory from the Introduction and then jumping to the needed explanations is how the book will mostly be used. Each chapter (other than Introduction) will contain between 5 to 10 "tips", each of which is a self-contained explanation with implementation detail generally demonstrated as pseudo code, or in cases where it makes sense, actual code.

The "VaMEx - Valles Marineris Explorer" initiative is part of the DLR Explorer Initiatives. As such it is an interdisciplinary research program funded by the DLR Space Administration aimed at developing new concepts, algorithms and hardware for swarm-based exploration of the Valles Marineris on Mars. This includes a hominid robotic platform (project VaMEx-VIPe), autonomous swarm navigation including ground vehicles and UAVs (project VaMEx-CoSMiC) that rely on a local positioning and landing system (project VaMEx-LAOLa), and orbital support (VaMEx-NavComNet) serving as a science data, telemetry and telecommand relay between Earth and the in-situ elements and providing near real-time position updates to the other elements. Real validation and verification tests for such complex navigation and exploration systems are difficult, expensive and time-consuming because they require the availability of hardware, realistic environments and software-in-the-loop. In this paper, we present VaMEx-VTB, a virtual testbed (VTB) that enables the verification and validation of such large and complex interdisciplinary research projects during very early phases. The basic idea of VaMEx-VTB is to provide a common software platform for all modules in combination with a sophisticated user definable computer simulation thereby it helps reducing expensive and time-consuming physical testing. Additionally, it can serve as an integration and discussion hub during the development process. The VTB allows users to configure various aspects of the test scenarios and the test environment, such as physical parameters, atmospheric conditions, or terrain features. This is essential especially for extraterrestrial planetary missions that are difficult to reconstruct on earth. Finally, a sophisticated graphical feedback, based on a state-of-the-art game engine, allows an easy and direct interaction of the engineers with the test case in the VTB. Our modular design based on ROS supports consistent data access for all components. So far, we have implemented a realistic simulation of the relevant environmental parameters and created an adjustable model of the Valles Marineris terrain, based on the HiRISE data. Additionally, the VTB synthesizes realistic sensor input for several algorithms running on the swarm elements. The modular design concept also qualifies the VTB to serve as a testing platform for other extraterrestrial missions in the future.

We present the first VR training simulator for hip replacement surgeries. We solved the main challenges of this task – high and stable forces during the milling process while simultaneously a very sensitive feedback is required – by using an industrial robot for the force output and the development of a novel massively parallel haptic rendering algorithm with support for material removal.

We present UnrealHaptics, a novel set of plugins that enable both 3-DOF and 6-DOF haptic rendering in the Unreal Engine 4. The core is the combination of the integration of a state-of-the-art collision detection library with support for very fast and stable force and torque computations and a general haptics library for the communication with different haptic hardware devices. Our modular and lightweight architecture makes it easy for other researchers to adapt our plugins to their own requirements. As a use case we have tested our plugin in a new asymmetric collaborative multiplayer game for blind and sighted people. The results show that our plugin easily meets the requirements for haptic rendering even in complex scenes.

We present a novel algorithm, called AstroGen, to procedurally generate highly detailed and realistic 3D meshes of small celestial bodies automatically. AstroGen gains it’s realism from learning surface details from real world asteroid data. We use a sphere packing-based metaball approach to represent the rough shape and a set of noise functions for the surface details. The main idea is to apply an optimization algorithm to adopt these representations to available highly detailed asteroid models with respect to a similarity measure. Our results show that our approach is able to generate a wide variety of different celestial bodies with very complex surface structures like caves and craters.

We contribute a new library, DynCam, for real-time, low latency, streaming point cloud processing with a special focus on telep- resence in VR. Our library combines several RGBD-images from multiple distributed sources to a single point cloud and transfers it through a network. This processing is organized as a pipeline that supports implicit multithreading. The pipeline uses functional reactive programming to describe transformations on the data in a declarative way. In contrast to previous libraries, DynCam is plat- form independent, modular and lightweight. This makes it easy to extend and allows easy integration into existing applications. We have prototypically implemented a telepresence application in the Unreal Engine. Our results show that DynCam outperforms competing libraries concerning latency as well as network traffic.

We consider the problem of finding automatically optimal positions of surgical lamps throughout the whole surgical procedure, where we assume that future lamps could be robotized. We propose a two-tiered optimization technique for the real-time autonomous positioning of those robotized surgical lamps. Typically, finding optimal positions for surgical lamps is a multi-dimensional problem with several, in part conflicting, objectives, such as optimal lighting conditions at every point in time while minimizing the movement of the lamps in order to avoid distractions of the surgeon. Consequently, we use multi-objective optimization (MOO) to find optimal positions in real-time during the entire surgery. Due to the conflicting objectives, there is usually not a single optimal solution for such kinds of problems, but a set of solutions that realizes a Pareto-front. When our algorithm selects a solution from this set it additionally has to consider the individual preferences of the surgeon. This is a highly non-trivial task because the relationship between the solution and the parameters is not obvious. We have developed a novel meta-optimization that considers exactly this challenge. It delivers an easy to understand set of presets for the parameters and allows a balance between the lamp movement and lamp obstruction. This meta-optimization can be pre-computed for different kinds of operations and it then used by our online optimization for the selection of the appropriate Pareto solution. Both optimization approaches use data obtained by a depth camera that captures the surgical site but also the environment around the operating table. We have evaluated our algorithms with data recorded during a real open abdominal surgery. It is available for use for scientific purposes. The results show that our meta-optimization produces viable parameter sets for different parts of an intervention even when trained on a small portion of it.

We present a novel method to define the penetration volume between a surface point cloud and arbitrary 3D CAD objects. Moreover, we have developed a massively-parallel algorithm to compute this penetration measure efficiently on the GPU. The main idea is to represent the CAD object's volume by an inner bounding volume hierarchy while the point cloud does not require any additional data structures. Consequently, our algorithm is perfectly suited for streaming point clouds that can be gathered online via depth sensors like the Kinect. We have tested our algorithm in several demanding scenarios and our results show that our algorithm is fast enough to be applied to 6-DOF haptic rendering while computing continuous forces and torques.

We present a novel statistical shape descriptor for arbitrary three-dimensional shapes as a six-dimensional feature for generic classification purposes. Our feature parameterizes the complete geometrical relation of the global shape and additionally considers local dissimilarities while being invariant to the shape appearance. Our approach allows the classification of large-scale shapes with only small local dissimilarities. Our feature can be easily quantized and mapped into a histogram, which can be used for efficient and effective classification. We take advantage of GPU processing in order to efficiently compute our invariant local shape descriptor feature even for large-scale shapes. Our synthetic benchmarks show that our approach outperforms state-of-the-art methods for local shape dissimilarity classification. In general, it yields robust and promising recognition rates even for noisy data.

We define a novel geometric predicate and a class of objects that enables us to prove a linear bound on the number of intersecting polygon pairs for colliding 3D objects in that class. Our predicate is relevant both in theory and in practice: it is easy to check and it needs to consider only the geometric properties of the individual objects – it does not depend on the configuration of a given pair of objects. In addition, it characterizes a practically relevant class of objects: we checked our predicate on a large database of real-world 3D objects and the results show that it holds for all but the most pathological ones. Our proof is constructive in that it is the basis for a novel collision detection algorithm that realizes this linear complexity also in practice. Additionally, we present a parallelization of this algorithm with a worst-case running time that is independent of the number of polygons. Our algorithm is very well suited not only for rigid but also for deformable and even topology-changing objects, because it does not require any complex data structures or pre-processing. We have implemented our algorithm on the GPU and the results show that it is able to find in real-time all colliding polygons for pairs of deformable objects consisting of more than 200k triangles, including self-collisions.

Currently, interest in space missions to small bodies (e.g., asteroids) is increasing, both scientifically and commercially. One of the important aspects of these missions is to test the navigation, guidance, and control algorithms. The most cost and time efficient way to do this is to simulate the missions in virtual testbeds. To do so, a physically-based simulation of the small bodies' physical properties is essential. One of the most important physical properties, especially for landing operations, is the gravitational field, which can be quite irregular, depending on the shape and mass distribution of the body. In this paper, we present a novel algorithm to simulate gravitational fields for small bodies like asteroids. The main idea is to represent the small body's mass by a polydisperse sphere packing. This allows for an easy and efficient parallelization. Our GPU-based implementation outperforms traditional methods by more than two orders of magnitude while achieving a similar accuracy.

We present a novel approach for approximating objective functions in arbitrary deterministic and stochastic multi-objective blackbox simulations. Usually, simulated-based optimization approaches require pre-defined objective functions for optimization techniques in order to find a local or global minimum of the specified simulation objectives and multi-objective constraints. Due to the increasing complexity of state-of-the-art simulations, such objective functions are not always available, leading to so-called blackbox simulations. In contrast to existing approaches, we approximate the objective functions and design space for deterministic and stochastic blackbox simulations, even for convex and concave Pareto fronts, thus enabling optimization for arbitrary simulations. Additionally, Pareto gradient information can be obtained from our design space approximation. Our approach gains its efficiency from a novel gradient-based sampling of the parameter space in combination with a density-based clustering of sampled objective function values, resulting in a B-spline surface approximation of the feasible design space.

We present a novel knowledge discovery approach for automatic feasible design space approximation and parameter optimization in arbitrary multiobjective blackbox simulations. Our approach does not need any supervision of simulation experts. Usually simulation experts conduct simulation experiments for a predetermined system specification by manually reducing the complexity and number of simulation runs by varying input parameters through educated assumptions and according to prior defined goals. This leads to a error-prone trial-and-error approach for determining suitable parameters for successful simulations. In contrast, our approach autonomously discovers unknown relationships in model behavior and approximates the feasible design space. Furthermore, we show how Pareto gradient information can be obtained from this design space approximation for state-of-the-art optimization algorithms. Our approach gains its efficiency from a novel spline-based sampling of the parameter space in combination within novel forest-based simulation dataflow analysis. We have applied our new method to several artificial and real-world scenarios and the results show that our approach is able to discover relationships between parameters and simulation goals. Additionally, the computed multiobjective solutions are close to the Pareto front.

We present a new graph-based approach called GraphPool for the generation, management and distribution of simulation states for 3D simulation applications. Currently, relational databases are often used for this task in simulation applications. In contrast, our approach combines novel wait-free nested hash map techniques with traditional graphs which results in a schema-less, in-memory, highly efficient data management. Our GraphPool stores static and dynamic parts of a simulation model, distributes changes caused by the simulation and logs the simulation run. Even more, the GraphPool supports sophisticated query types of traditional relational databases. As a consequence, our GraphPool overcomes the associated drawbacks of relational database technology for sophisticated 3D simulation applications. Our GraphPool has several advantages compared to other state-of-the-art decentralized methods, such as persistence for simulation state over time, object identification, standardized interfaces for software components as well as a consistent world model for the overall simulation system. We tested our approach in a synthetic benchmark scenario but also in real-world use cases. The results show that it outperforms state-of-the-art relational databases by several orders of magnitude.

We present the first accessible game that allows a fair competition between sighted and blind people in a shared virtual 3D environment.We use an asymmetric setup that allows touchless interaction via Kinect, for the sighted player, and haptic, wind, and surround audio feedback, for the blind player. We evaluated our game in an in-the-wild study. The results show that our setup is able to provide a mutually fun game experience while maintaining a fair winning chance for both players. Based on our study, we also suggest guidelines for future developments of games for visually impaired people that could help to further include blind people into society.

In the past, the Entity-Component-System (ECS) pattern has become a major design pattern used in modern architectures for Realtime Interactive Systems (RIS). In this paper we introduce high performance wait-free hash maps for the System access of Components within the ECS pattern. This allows non-locking read and write operations, leading to a highly responsive low-latency data access while maintaining a consistent data state. Furthermore, we present centralized as well as decentralized approaches for reducing the memory demand of these memory-intensive wait-free hash maps for diverse RIS applications. Our approaches gain their efficiency by Component-wise queues which use atomic markup operations for fast memory deletion. We have implemented our new method in a current RIS and the results show that our approach is able to efficiently reduce the memory usage of wait-free hash maps very effectively by more than a factor of ten while still maintaining their high performance. Furthermore, we derive best practices from our numerical results for different use cases of wait-free hash map memory management in diverse RIS applications.

We present a novel framework for the simultaneous development for different massively parallel platforms. Currently, our framework supports CUDA and OpenCL but it can be easily adapted to other programming languages. The main idea is to provide an easy-to-use abstraction layer that encapsulates the calls of own parallel device code as well as library functions. With our framework the code has to be written only once and can then be used transparently for CUDA and OpenCL. The output is a single binary file and the application can decide during run-time which particular GPU-method it will use. This enables us to support new features of specific platforms while maintaining compatibility. We have applied our framework to a typical project using CUDA and ported it easily to OpenCL. Furthermore we present a comparison of the running times of the ported library on the different supported platforms.

We present a novel method for the autonomous positioning of surgical lamps in open surgeries. The basic idea is to use an inexpensive depth camera to track all objects and the surgical staff and also generate a dynamic online model of the operation situs. Based on this information, our algorithms continuously compute the optimal positions for all surgical lamps. These positions can then be communicated to robotic arms so that the lamps mounted on their end effectors will move autonomously. This will ensure optimal lighting of the operation situs at all times, while avoiding occlusions and shadows from obstacles. We tested our algorithm in a VR simulation using real-world depth camera data that was recorded during a real abdominal operation. Our results show that our method is robust and can ensure close-to-optimal lighting conditions in real-world surgeries with an update rate of 20 Hz.

Modelling, simulation, and optimization play a crucial role in the development and testing of autonomous vehicles. The ability to compute, test, assess, and debug suitable configurations reduces the time and cost of vehicle development. Until now, engineers are forced to manually change vehicle configurations in virtual testbeds in order to react to inappropriate simulated vehicle performance. Such manual adjustments are very time consuming and are also often made ad-hoc, which decreases the overall quality of the vehicle engineering process. In order to avoid this manual adjustment as well as to improve the overall quality of these adjustments, we present a novel comprehensive approach to modelling, simulation, and optimization of such vehicles. Instead of manually adjusting vehicle configurations, engineers can specify simulation goals in a domain specific modelling language. The simulated vehicle performance is then mapped to these simulation goals and our multi-agent system computes for optimized vehicle configuration parameters in order to satisfy these goals. Consequently, our approach does not need any supervision and gives engineers visual feedback of their vehicle configuration expectations. Our evaluation shows that we are able to optimize vehicle configuration sets to meet simulation goals while maintaining real-time performance of the overall simulation.

We present a novel concurrency control mechanism for collaborative massively parallel virtual environments that allows an arbitrary amount of components to exchange data with very little synchronisation overhead. The approach taken here is to maintain the shared world state of the complete virtual environment in a global key-value pool. Our novel method does not use any locking mechanism. Instead it allows wait-free data access for all concurrent components for both, reading and writing operations. This guarantees a highly responsive low-latency data access while keeping a consistent system state for all users and system components. Nevertheless, our approach is perfectly scalable even for massive multi-user scenarios. We provide a number of benchmarks in this paper, and the results show an almost constant running time, independent of the number of concurrent users. Moreover, our approach outperforms previous concurrency control systems significantly by more than an order of magnitude.

Ordinary bounding volume hierarchy (BVH) construction algorithms create BVHs that approximate the boundary of the objects. In this paper, we present a BVH construction that instead approximates the volume of the objects with successively finer levels. It is based on Batch Neural Gas (BNG), a clustering algorithm that is known from machine learning. Additionally, we present a novel massively parallel version of this BNG-based hierarchy construction that runs completely on the GPU. It reduces the theoretical complexity of the sequential algorithm from O(nlogn) to O(log2 n) and also our CUDA implementation outperforms the CPU version significantly in practice.

We present a novel massively-parallel algorithm that allows real-time distance computations between arbitrary 3D objects and unstructured point cloud data. Our main application scenario is collision avoidance for robots in highly dynamic environments that are recorded via a Kinect, but our algorithm can be easily generalized for other applications such as virtual reality. Basically, we represent the 3D object by a bounding volume hierarchy, therefore we adopted the Inner Sphere Trees data structure, and we process all points of the point cloud in parallel using GPU optimized traversal algorithms. Additionally, all parallel threads share a common upper bound in the minimum distance, this leads to a very high culling efficiency. We implemented our algorithm using CUDA and the results show a real-time performance for online captured point clouds. Our algorithm outperforms previous CPU-based approaches by more than an order of magnitude.

A central part of virtual reality systems and game engines is the generation, management and distribution of all relevant world states. In modern interactive graphic software systems usually many independent software components need to communicate and exchange data. Standard approaches suffer the n 2 problem because the number of interfaces grows quadratically with the number of component functionalities. Such many-to-many architectures quickly become unmaintainable, not to mention latencies of standard concurrency control mechanisms. We present a novel method to manage concurrent multithreaded access to shared data in virtual environments. Our highly efficient low-latency and lightweight architecture is based on a new wait-free hash map using key-value pairs. This allows us to reduce the traditional many-to-many problem to a simple many-to-one approach. Our results show that our framework outperforms by more than two orders of magnitude standard lock-based but also modern lock-free methods significantly.

Als etablierte Plattform für den Informations- und Ideenaustausch der deutschsprachigen VR/AR-Szene bietet der Workshop den idealen Rahmen, aktuelle Ergebnisse und Vorhaben aus Forschung und Entwicklung im Kreise eines fachkundigen Publikums zur Diskussion zu stellen.

We prove that the maximum number of intersecting pairs spheres between two sets of polydisperse sphere packings is linear in the worst case. This observation is the basis for a new collision detection algorithm. Our new approach guarantees a linear worst case running time for arbitrary 3D objects. Additionally, we present a parallelization of our new algorithm that runs in constant time, even in the worst case. Consequently, it is perfectly suited for all time-critical environments that allow only a fixed time budget for finding collision. Our implementation using CUDA shows collision detection at haptic rates for complex objects.

Polydisperse sphere packings are a new and very promising data representation for several fundamental problems in computer graphics and VR such as collision detection and deformable object simulation. In this paper we present acceleration techniques to compute such sphere packings for arbitrary 3D objects efficiently on the GPU. To do that, we apply different refinement methods for adaptive grids. Our results show a significant speed-up compared to existing approaches.

We present new geometric data structures for collision detection and more, including: Inner Sphere Trees - the first data structure to compute the peneration volume efficiently. Protosphere - an new algorithm to compute space filling sphere packings for arbitrary objects. Kinetic AABBs - a bounding volume hierarchy that is optimal in the number of updates when the objects deform. Kinetic Separation-List - an algorithm that is able to perform continuous collision detection for complex deformable objects in real-time. Moreover, we present applications of these new approaches to hand animation, real-time collision avoidance in dynamic environments for robots and haptic rendering, including a user study that exploits the influence of the degrees of freedom in complex haptic interactions. Last but not least, we present a new benchmarking suite for both, peformance and quality benchmarks, and a theoretic analysis of the running-time of bounding volume-based collision detection algorithms.

We present the results of a comprehensive user study that evaluates the influence of the degrees of freedom on the users' performance in complex bi-manual haptic interaction tasks. To do that, we have developed a novel multi-player game that allows the qualitative as well as the quantitative evaluation of different force-feedback devices simultaneously. The game closely resembles typical tasks arising in tele-operation scenarios or virtual assembly simulations; thus, the results of our user study apply directly to real-world industrial applications. The game is based on our new haptic workspace that supports high fidelity, two-handed multi-user interactions in scenarios containing a large number of dynamically simulated rigid objects; moreover, it works independent of the objects' polygon count. The results of our user study show that 6 DOF forcefeedback devices outperform 3 DOF devices significantly, both in user perception and in user performance.

We present a haptic workspace that allows high fidelity two-handed multi-user interactions in scenarios containing a large number of dynamically simulated rigid objects and a polygon count that is only limited by the capabilities of the graphics card. Based in this workspace we present a novel multiplayer game that supports qualitative as well as quantitative evaluation of different haptic devices in demanding haptic interaction tasks.

Collision detection between rigid objects plays an important role in many fields of robotics and computer graphics, e.g. for path-planning, haptics, physically-based simulations, and medical applications.

This chapter contributes the following novel ideas to the area of collision detection:

We present a new algorithm that is able to efficiently compute a space filling sphere packing for arbitrary objects. It is independent of the object's representation (polygonal, NURBS, CSG,...); the only precondition is that it must be possible to compute the distance from any point to the surface of the object. Moreover, our algorithm is not restricted to 3D but can be easily extended to higher dimensions.

The basic idea is very simple and related to prototype based approaches known from machine learning. This approach directly leads to a parallel algorithm that we have implemented using CUDA. As a byproduct, our algorithm yields an approximation of the object's medial axis that has applications ranging from path-planning to surface reconstruction.

A benchmarking suite for rigid object collision detection and collision response schemes. The proposed benchmarking suite can evaluate both the performance as well as the quality of the collision response. The former is achieved by densely sampling the configuration space of a large number of highly detailed objects; the latter is achieved by a novel methodology that comprises a number of models for certain collision scenarios. With these models, we compare the force and torque signals both in direction and magnitude.

Our device-independent approach allows objective predictions for physically-based simulations as well as 6-DOF haptic rendering scenarios. In the results, we show a comprehensive example application of our benchmarks comparing two quite different algorithms utilizing our proposed benchmarking suite. This proves empirically that our methodology can become a standard evaluation framework.

Based on our new geometric data structure, the inner sphere trees, we present a fast and stable uniform algorithm for proximity and penetration volume queries between watertight objects at haptic rates.

Moreover, we present a multi-threaded version of the penetration volume computation for time-critical haptic rendering that is based on separation lists and the novel notion of expected overlapping volumes. Finally, we show how to use the penetration volume to compute continuous contact forces and torques that enable a stable rendering of 6-DOF penalty-based distributed contacts.

Since the visual feedback and effects of today's games have become extremely mature, it will be more and more important for games to provide realistic feedback to other senses, such as our haptic sense. On the hardware side, this has become possible in recent years by the advent of first inexpensive haptic devices on the consumer market, such as the Falcon from Novint. Research on force-feedback devices and algorithms has been done over 10 years, and has only fairly recently been introduced to games.

However, while there is a large body of research on how to render forces given a collision and its contact information, the computation of the latter for massive models is still a challenge. First of all, this is due to the much higher effort to compute contact information. Second, this is due to the update rates that are necessary for haptic rendering, which need to be much higher than for visual rendering, i.e., 250-1000 Hz. And third, defining the contact information such that continuous contact forces can be derived is not always obvious.

Therefore, one of the major challenges in haptic rendering for games is the computation of continuous forces at haptic rates. A solution to this challenge can also be utilized to do physically-based simulation of rigid bodies, which has become increasingly popular in games over the past few years.

In this paper, we take advantage of the fact that in rendering haptic forces, as well as in most real-time applications that involve physically-based simulation, an absolutely correct determination of the forces acting on the virtual objects is not necessary.

We present a novel geometric data structure for approximate collision detection at haptic rates between rigid objects. Our data structure, which we call inner sphere trees, supports different kinds of queries, namely, proximity queries and a new method for interpenetration computation, the penetration volume, which is related to the water displacement of the overlapping region and, thus, corresponds to a physically motivated force. The main idea is to bound objects from the inside with a set of non-overlapping spheres. Based on such sphere packings, a "inner bounding volume hierarchy" can be constructed. In order to do so, we propose to use an AI clustering algorithm, which we extend and adapt here. The results show performance at haptic rates both for proximity and penetration volume queries for models consisting of hundreds of thousands of polygons.

In this paper, we present a benchmarking suite that allows a systematic comparison of pairwise static collision detectionalgorithms for rigid objects.The benchmark generates a number of positions and orientations for a predefined distance. We implemented the benchmarking procedure and compared a wide number of freely available collision detection algorithms.

In this paper, we propose a model to estimate the expected running time of hierarchical collision detection that utilizes AABB trees, which are a frequently used type of bounding volume (BV). We show that the average running time for the simultaneous traversal of two binary AABB trees depends on two characteristic parameters: the overlap of the root BVs and the BV diminishing factor within the hierarchies. With this model, we show that the average running time is in O(n) or even in O(logn) for realistic cases. Finally, we present some experiments that confirm our theoretical considerations. We believe that our results are interesting not only from a theoretical point of view, but also for practical applications, e. g., in time-critical collision detection scenarios where our running time prediction could help to make the best use of CPU time available.

We present novel algorithms for updating bounding volume hierarchies of objects undergoing arbitrary deformations. Therefore, we introduce two new data structures, the kinetic AABB tree and the kinetic BoxTree. The event-based approach of the kinetic data structures framework enables us to show that our algorithms are optimal in the number of updates. Moreover, we show a lower bound for the total number of BV updates, which is independent of the number of frames. We used our kinetic bounding volume hierarchies for collision detection and performed a comparison with the classical bottom-up update method. The results show that our algorithms perform up to ten times faster in practically relevant scenarios.

We present a new acceleration scheme for continuous collision detection of objects under arbitrary deformations. Both pairwise and self collision detection are presented. This scheme is facilitated by a new acceleration data structure, the kinetic separation list. The event-based approach of our kinetic separation list enables us to transform the continuous problem into a discrete one. Thus, the number of updates of the bounding volume hierarchies as well as the number of bounding volume checks can be reduced significantly. We performed a comparison of our kinetic approaches with the classical swept volume algorithm. The results shows that our algorithm performs up to fifty times faster in practically relevant scenarios.