RESEARCH

2001

The Virtual Showcase

Zoom: Click on Image Intuitive access to information in habitual realworld environments is a challenge for information technology. An important question is how can we enhance established and well-functioning everyday environments rather than replace them by virtual environments (VEs)? Augmented reality (AR) technology has a lot of potential in this respect because it augments realworld environments with computer-generated imagery. Today, most AR systems use see-through head-mounted displays, which share most of the disadvantages of other head-attached display devices. We introduce a new projection-based AR display system—the Virtual Showcase. The Virtual Showcase has the same form factor as a real showcase, making it compatible with traditional museum displays. Real scientific and cultural artifacts are placed inside the Virtual Showcase allowing their 3D graphical augmentation. Inside the Virtual Showcase, virtual representations and real artifacts share the same space providing new ways of merging and exploring real and virtual content. Solely virtual exhibits may also be displayed. The virtual part of the showcase can react in various ways to a visitor, which provides the possibility for intuitive interaction with the displayed content. Another interesting aspect of our system is its support for two to four simultaneously tracked users looking at the Virtual Showcase from different sides. This feature allows the collaborative exploration of artifacts shown in the Virtual Showcase. These interactive showcases contribute to ambient intelligent landscapes, where the computer acts as an intelligent server in the background and visitors can focus on exploring the exhibited content rather than on operating computers. The Virtual Showcase consists of two main parts: a convex assembly of halfsilvered mirrors and a graphics display. So far, we’ve built Virtual Showcases with two different mirror configurations. Our first prototype consists of four halfsilvered mirrors assembled as a truncated pyramid. Our second prototype uses a single mirror sheet to form a truncated cone. We placed these mirror assemblies on top of a projection screen. Users can see real objects inside the showcase through the half-silvered mirrors merged with the graphics displayed on the projection screen. We illuminated the showcases’ contents with a controllable light source while presenting view-dependent stereoscopic graphics to the observer. For our current prototypes, we use standard shutter glasses controlled by infrared emitters. Head tracking is accomplished using an electromagnetic tracking device. Our pyramid shaped prototype supports up to four viewers simultaneously looking at the showcase from four different sides. Our cone-shaped prototype provides a seamless surround view onto the displayed artifact.

Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. The Virtual Showcase IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55, 2001 Movie 1 (~58MB) Movie 2 (~63MB)

The Extended Virtual Table

Zoom: Click on Image We describe a prototype of an optical extension for table-like rear-projection systems -the Extended Virtual Table. A large half-silvered mirror is used as the optical combiner to unify a virtual and a real workbench, whereby the shortcomings that are related to the Reflective Pad and Transflective Pad (i.e., indirect line-of-sight, limited field of view, and tracking distortion) are reduced. The virtual workbench has been enabled to display computer graphics beyond its projection boundaries and to combine virtual environments with the adjacent real world. A variety of techniques is described that allow indirect interaction with virtual objects through the mirror. Systems, such as the Extended Virtual Table, approach a conceptual and technical extension of traditional Virtual Reality by means of Augmented Reality (xVR), and a seamless integration of such technology into habitual work environments. Furthermore, optical distortions that are caused by the half-silvered mirror combiner and the projector, as well as non-optical distortions caused by the tracking device are analyzed and appropriate compensation methods are described.

Bimber, O., Encarnação, L.M. and Branco, P. The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems Presence: Teleoperators and Virtual Environments, vol.10, no. 6, pp. 613-631, 2001 Movie (~15MB)

Projection-Based Augmented Engineering

Zoom: Click on Image Traditional Augmented Reality (AR) displays, such as head-mounted devices entail a number of technological and ergonomic drawbacks. Their limited resolution, the restricted field-of-view, the visual perception issues that are due to the fixed focal length caused by a constant and head-attached image plane, the increased incidence of discomfort provoked by simulation sickness and their cumbersomeness prevent their usage in a number of application areas. To overcome some of these drawbacks, but also to open new application areas for AR, we propose a projectionbased AR concept that combines spatially aligned optical see-through elements (essentially half-silvered mirror beam-splitters) with off-the-shelf projection-based Virtual Reality displays. This concept offers possibilities to combine the advantages of both technologies: the well established projection-based Virtual Reality with the potentials of Augmented Reality. We describe the early stages of a new proof of-concept prototype that has been developed to extend the scope of projection-based AR towards the engineering domain.

Bimber, O., Stork, A., and Branco, P. Projection-based Augmented Engineering Human-Computer Interaction International (HCII’01), vol. 1, pp. 787-791, 2001 Movie (~15MB)

Seamless Integration of Virtual Reality into habitual Workplaces

Zoom: Click on Image Employing virtual reality (VR) technology, computer-based work environments are clearly evolving into a major component of the next-generation workplace. Over the past few years, VR has become a practical reality for many applications, thanks to a number of technology inventions. Among these has been the development of large-screen display systems that make use of stereoscopic projection. Systems such as the Virtual Table and Responsive Workbench  both employ a horizontal projection philosophy, whereas other devices, such as the Powerwall  and surround-screen projection systems (SSPS) like the CAVE , also employ vertical back-projection. Due to their effective application, these devices are gaining greater acceptance by the research and industrial user communities. Such devices offer a large field of view, brilliant image quality, high resolution and (especially the table-like devices) a promising compatibility with traditional workspaces. To support this integration, however, ergonomic and human-centered interaction techniques must be developed. We introduce ideas, proposed technologies, and initial results for a seamless integration of virtual reality in habitual workspaces, thus successively transforming them into efficient, high-tech work environments.

Bimber, O., Encarnação, L.M. and Stork, A. Seamless integration of virtual reality in habitual workplaces Journal for Industrial Science, Munich University of Technology, vol.55, no.2, pp. 103-112, 2001 Movie (~15MB)

2000

Transflective Props

Zoom: Click on Image We describe the use of a hand-held semi-transparent mirror to support augmented reality tasks with back-projection systems. This setup overcomes the problem of occlusion of virtual objects by real ones linked with such display systems. The presented approach allows an intuitive and effective application of immersive or semi-immersive virtual reality tasks and interaction techniques to an augmented surrounding space. Thereby, we use the tracked mirror as an interactive image-plane that merges the reflected graphics, which are displayed on the projection plane, with the transmitted image of the real environment. In our implementation, we also address traditional augmented reality problems, such as real-object registration and virtual-object occlusion. The presentation is complemented by a hypothesis of conceivable further setups that apply transflective surfaces to support a mixed reality (i.e., combined AR and VR) environment.

Bimber, O., Encarnação, L.M., and Schmalstieg, D. Augmented Reality with Back-Projection Systems using Transflective Surfaces Computer Graphics Forum (proceedings of EUROGRAPHICS 2000 - EG’2000), vol. 19, no. 3, pp. 161-168, 2000 Movie (~19MB)

Reflective Props

Zoom: Click on Image We introduce the idea of using real mirrors in combination with rear-projection systems for the purpose of interacting with and navigating through the displayed information. Subsequently a derived application is described. For this, we use a hand-held planar mirror and address two fundamental problems of applying head tracking with rear-projection planes: the limited viewing volume of these environments and their incapability of simultaneously supporting multiple observers. Furthermore, we describe the possibility of combining a reflective pad with a transparent one, thus introducing a complementary tool for interaction and navigation.

Bimber, O., Encarnação, L.M., and Schmalstieg, D. Real Mirrors Reflecting Virtual Worlds In proceedings of IEEE Virtual Reality (IEEE VR’00), pp. 21-28, 2000

Sketch-Based and Multi-Modal Interaction

Zoom: Click on Image Leonardo da Vinci's drawings of machines and other objects illustrate one of the most fundamental purposes of sketches: the ability to communicate design and functionality to others. Nowadays, it is widely accepted that sketching is a form of critical, reflective dialog that handles communication on one or more different levels of abstraction simultaneously. Various approaches have been taken to support this kind of dialog between humans and computers, and to build human-computer interfaces that are able to interpret such freehand sketches for different purposes. In this context, the creation or reconstruction of 3D objects from 2D sketches is of major concern in many application areas. This so-called 'pencil-and-paper' approach is used for rapidly designing approximate threedimensional scenes. While some systems analyze the orthographic or perspective projections to reconstruct 3D shapes that, based on psychological assumptions, are most plausible to the human observer, others interpret 2D gestures while the objects are sketched. Within the last decade, the conceptual design phase has been increasingly supported by sketch systems that allow the expression of ideas on a computer-aided, but still human-centered basis. However, putting an emphasis on sketching, most of these systems are sealed off from real-world applications rather than being generally applicable as components. We introduce a framework for sketch-based interaction within three-dimensional virtual environments. We describe each layer of the framework using illustrative examples depicting its realization. Furthermore, we want to present a variety of domain-specific applications of sketching within 3D virtual environments based on our framework instead of implementing yet another application for sketching.

Bimber, O., Encarnação, L.M. and Stork, A. A Multi-Layered Architecture for Sketch-based Interaction within Three-dimensional Virtual Environments Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 24, no. 6, pp. 851-867, 2000

Motion-Based Gesture Recognition

Zoom: Click on Image The method described in this paper recognizes previously learned gestures which the system was taught by performing them. Any kind of 6DOF input device (e.g. data glove, visual systems, space mouse etc.) can be used to gather the motion data that carries out the gesture. The reliability of the recognition process can be improved by repeating the same gestures several times and correcting wrong recognition results. This extends the system’s knowledge. Once learned, the system translates the recognized gestures into (for a computer) identifiable objects (numbers, strings, events, etc.) that can be further processed. The system’s advantages are its usability (e.g. with 2D or 3D input devices or in combination with finger status information, etc.), the minimum of information needed to recognize a gesture, and, consequently, the high speed of its scanning and comparison process.

Bimber, O. Continuous 6D Gesture Recognition: A Fuzzy-Logic Approach In proceedings of 7-th International Conference in Central Europe on Computer Graphics, Visualization and Interactive Digital Media (WSCG'99), vol. 1, pp. 24-30, 1999