RESEARCH

2003

Superimposing Pictorial Artwork with Projected Imagery

Zoom: Click on Image Pictorial artwork, such as paintings and drawings, can tell interesting stories. The capabilities of museums to communicate such and other information, however, are clearly limited. Text legends and audio guides can mediate facts, but offer little potential for presenting visual content, such as embedded illustrations, pictures, animations, and interactive elements. Also due to the difficult economic situation, edutainment is becoming an important factor for museums. By applying new media technologies – such as computer graphics, virtual reality and augmented reality– exhibit oriented information might be communicated more effectively, but certainly in a more exciting way. We describe a novel technological approach, a mathematical model, a real-time rendering algorithm and examples of presentation techniques for integrating almost any kind of visual information directly into pictorial artwork. It allows displaying such information while keeping the observers’ attention on the original artifact, and does not require additional screens. Special thanks to the Museum het Rembrandthuis, Amsterdam for their support.

Bimber, O., Coriand, F., Kleppe, A., Bruns, E., Zollmann, S., and Langlotz, T. Superimposing Pictorial Artwork with Projected Imagery IEEE MultiMedia, pp. 16-26, January-March issue 2005 Movie (Technology) (~24MB) DivX Codec Movie (Creation of Adam Demonstration) (~46MB) DivX Codec

The HoloStation

Zoom: Click on Image The Holographic Workstation (short: HoloStation) is a desktop display that allows the integration of interactive stereoscopic or auto-stereoscopic graphical elements into high-quality optical holograms (see below). Archaeologists, for instance, use optical holograms to archive and investigate ancient artefacts. The distribution of copy holograms enables scientists to perform their research without having access to the original artefacts or inaccurate replicas. Combining these static holograms with interactive computer graphics will allow them to integrate real-time simulation data or to perform experiments that require a direct user interaction, such as packing reconstructed soft-tissue into a hologram of a fossilized dinosaur skull. In addition, specialized interaction devices can simulate haptic feedback of the holographic and the graphical content while performing such interactive tasks. An entire collection of artefacts will fit into a single album of holographic recordings, while a light-box-like display can be used for visualization and interaction. The current HoloStation prototype can be used in combination with many different holograms, such as transmission and reflection types. It is a compact and flexible display that provides several forms of interaction via integrated touch-screen and real-time range scanner. Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support. More information on the HoloGraphics project.

Bimber, O. Augmenting Holograms IEEE Computer Graphics and Applications, September 2006 Bimber, O. Combining Optical Holograms with Interactive Computer Graphics IEEE Computer (cover feature), pp. 85-91, January 2004 Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics The Holographer.org, April 2004 Bimber, O. Combining Optical Holograms with Interactive Computer Graphics Holography, SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005 HoloGraphics Project Web-Site Movie (~15MB)

Combining Optical Holograms with Interactive Computer Graphics

Zoom: Click on Image A hologram is a photometric emulsion that records interference patterns of coherent light. The recording itself stores amplitude, wavelength and phase information of light waves. In contrast to simple photographs (which can record only amplitude and wavelength information), holograms have the ability to reconstruct complete optical wavefronts. This results in a three-dimensional appearance of the captured scenery which is observable from different perspectives. Today, many applications for optical holograms exist. Examples include interferometry, copy protections, data storage and holographic optical elements. Optical holograms, however, are static and lack in interactivity. Electro-holography aims at the computer-based generation and display of holograms in real time. Since a massive amount of data has to be processed, transmitted and stored to create holograms, today’s computer technology still sets the limits of electro-holography. To overcome some of the performance issues, advanced reduction and compression methods have been developed. This results in electro-holograms that are interactive, but small, low-resolution and pure in color. A novel approach has been proposed that combines holography, autostereoscopy and projection technology to integrate interactive graphical elements into high-quality optical holograms. Beside the conceptual idea and the optical solutions, efficient rendering techniques and a proof-of-concept prototype are presented. Furthermore, potential application areas, such as museums and scientific visualization are outlined. Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support. More information on the HoloGraphics project.

Bimber, O. Augmenting Holograms IEEE Computer Graphics and Applications, September 2006 Bimber, O. Combining Optical Holograms with Interactive Computer Graphics IEEE Computer (cover feature), pp. 85-91, January 2004 Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics The Holographer.org, April 2004 Bimber, O. Combining Optical Holograms with Interactive Computer Graphics Holography, SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005 HoloGraphics Project Web-Site Movie (~15MB)

Consistent Illumination within Augmented Environments

Zoom: Click on Image To achieve a consistent lighting situation between real and virtual environments is important for convincing augmented reality (AR) applications. A rich pallet of algorithms and techniques have been developed that match illumination for video- or image-based augmented reality. However, very little work has been done in this area for optical see-through AR. We believe that the optical see-through concept is currently the most advanced technological approach to provide an acceptable level of realism and interactivity. Methods have been developed which create a consistent illumination between real and virtual components within an optical see-through environment – such as the Virtual Showcase. Combinations of video projectors and cameras are applied to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. For diffuse objects, the capturing process can also benefit from hardware acceleration – supporting dynamic update rates. To handle indirect lighting effects (like color bleeding) an off-line radiosity procedure is outlined that consists of multiple rendering passes. For direct lighting effects (such as simple shading, shadows and reflections) hardware accelerated techniques are described which allow to achieve interactive frame rates. The reflectance information is used in addition to solve a main problem of a previously introduced technique which creates consistent occlusion effects for multiple users within such environments.

Bimber, O.,Grundhöfer, A., Wetzstein, G., and Knödel, S.  Consistent Illumination within Optical See-Through Augmented Environments In proceedings of International Symposium on Mixed and Augmented Reality(ISMAR’03), pp. 198-207, 2003 Movie (~16MB) DivX Codec

Augmented Reality Digital Storytelling

Zoom:: Click on Image Interactive digital storytelling techniques are recently being applied in combination with new media forms, such as virtual reality (VR) and augmented reality (AR). Thereby the technological progress that is being made within these areas allows shifting interactive digital storytelling more and more into the third dimension and into the physical world. One of the main advantages of this transition is the possibility to communicate information more effectively with digital means by telling stories that can be experienced directly within a real environment or in combination with physical objects. The user experience is thus transformed from relating different pieces of information to one another to ‘living through’ the narrative. The perceptual quality and the unique aura of a real environment (e.g., a historical site) or object (e.g., an ancient artifact) cannot be simulated by today’s technology. Thus it is not possible to substitute them with virtual or electronic copies without them losing their flair of originality. This circumstance can be a crucial reason for using augmented reality as a technological basis for interactive digital storytelling.  Several research groups apply personal displays, such as head-mounted displays (HMDs) to realize AR-based new media experiences. These all-purpose display types, however, have to face technological problems that up until today have not been sufficiently solved1. These shortcomings cause a substantial credibility gap if a certain level of realism is required. We want to discuss how an application-customized type of augmented reality display –the Virtual Showcase– overcomes most of these technological shortcomings, and how it can be used as a new platform for digital storytelling. To underline our statements we discuss and evaluate a case study that focuses on using AR digital storytelling to communicate scientific information to a novice audience in a museum context. We have introduced the application that led to our case study in a previous publication. In this paper, we want to describe the technical components that have been developed to realize this application. In addition, we present a first user feedback and illustrate our current efforts of improvement.

Bimber,O.,  Encarnação, L.M., and Schmalstieg, D. The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling In proceedings of IPT/EGVE 2003 workshop, pp. 87-95, 2003 Movie(~95MB)

Real-Time Image Warping for Curved Mirror Displays

Zoom: Click on Image Displays which require a non-linear pre-distortion to present undistorted gfx on non-planar surfaces, or to neutralize optical distortion usually apply multi-pass rendering techniques. Projective textures or uniform grids are used to deform the image generated during the first pass before it is displayed as a texture map during the final pass. However, these approaches do not consider the error that is generated from a piecewise linear texture interpolation to adapt the underlying geometry. Curved mirror displays, for instance, that stereoscopically produce three-dimensional gfx generally don't pre-distort the graphics before they are displayed. Yet, some systems apply additional optics (such as lenses) to stretch or undistort the reflected image. But these devices constrain the observer to a single point of view or to very restricted viewing zones. For the Virtual Showcase mirror display, however, a view-dependent rendering is required to support freely moving observers. The existing image warping algorithms for curved Virtual Showcases predistort a uniform image grid and consequently do not consider the local error that is generated from a piecewise linear texture interpolation. These algorithms can only produce an acceptable image quality within a significantly large amount of rendering time. The algorithm that is presented improves the uniform image warping techniques for curved Virtual Showcases by defining an appropriate error metric and implementing a quadtree-based selective refinement method that generates adapted local levels of detail. It approaches to produce a predefined image quality within a minimum amount of rendering time. While display specific details of our algorithm are explained based on the Virtual Showcase display, its general functionality is valid for other non-linear displays -such as for curved projection-based systems.

Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L.M. Real-Time View-Dependent Image Warping to correct Non-Linear Distortion for Curved Virtual Showcase Displays In Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 27, no. 4, pp. 512-528, 2003 Movie (~36MB)