Research
DepthLab: Real-time 3D interaction with depth maps for mobile augmented reality
Mobile devices with passive depth sensing capabilities are ubiquitous, and recently active depth sensors have become available on some tablets and AR/VR devices. Although real-time depth data is accessible, its rich value to mainstream AR applications has been sorely under-explored. Adoption of depth-based UX has been impeded by the complexity of performing even simple operations with raw depth data, such as detecting intersections or constructing meshes. In this paper, we introduce DepthLab, a software library that encapsulates a variety of depth-based UI/UX paradigms, including geometry-aware rendering (occlusion, shadows), surface interaction behaviors (physics-based collisions, avatar path planning), and visual effects (relighting, 3D-anchored focus and aperture effects). We break down the usage of depth into localized depth, surface depth, and dense depth, and describe our real-time algorithms for interaction and rendering tasks. We present the design process, system, and components of DepthLab to streamline and centralize the development of interactive depth features. We have open-sourced our software at https://github.com/googlesamples/arcore-depth-lab to external developers, conducted performance evaluation, and discussed how DepthLab can accelerate the workflow of mobile AR designers and developers. With DepthLab we aim to help mobile developers to effortlessly integrate depth into their AR experiences and amplify the expression of their creative vision.
Ruofei Du, Eric Turner, Maksym Dzitsiuk, Luca Prasso, Ivo Duarte, Jason Dourgarian, Joao Afonso, Jose Pascoal, Josh Gladstone, Nuno Cruces, Shahram Izadi, Adarsh Kowdle, Konstantine Tsotsos, David Kim — ACM UIST, Oct 2020
Articulated distance fields for ultra-fast tracking of hands interacting
The state of the art in articulated hand tracking has been greatly advanced by hybrid methods that fit a generative hand model to depth data, leveraging both temporally and discriminatively predicted starting poses. In this paradigm, the generative model is used to define an energy function and a local iterative optimization is performed from these starting poses in order to find a good local minimum (i.e. a local minimum close to the true pose). Performing this optimization quickly is key to exploring more starting poses, performing more iterations and, crucially, exploiting high frame rates that ensure that temporally predicted starting poses are in the basin of convergence of a good local minimum. At the same time, a detailed and accurate generative model tends to deepen the good local minima and widen their basins of convergence. Recent work, however, has largely had to trade-off such a detailed hand model with one that facilitates such rapid optimization. We present a new implicit model of hand geometry that mostly avoids this compromise and leverage it to build an ultra-fast hybrid hand tracking system. Specifically, we construct an articulated signed distance function that, for any pose, yields a closed form calculation of both the distance to the detailed surface geometry and the necessary derivatives to perform gradient based optimization. There is no need to introduce or update any explicit correspondences yielding a simple algorithm that maps well to parallel hardware such as GPUs. As a result, our system can run at extremely high frame rates (e.g. up to 1000fps). Furthermore, we demonstrate how to detect, segment and optimize for two strongly interacting hands, recovering complex interactions at extremely high framerates. In the absence of publicly available datasets of sufficiently high frame rate, we leverage a multiview capture system to create a new 180fps dataset of one and two hands interacting together or with objects.
Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, Shahram Izadi — ACM TOG, Nov 2017
Holoportation: Virtual 3D Teleportation in Real-time
Holoportation is a new type of 3D capture technology that allows high quality 3D models of people to be reconstructed, compressed, and transmitted anywhere in the world in real-time. When combined with mixed reality displays such as HoloLens, this technology allows users to see and interact with remote participants in 3D as if they are actually present in their physical space. Communicating and interacting with remote users becomes as natural as face to face communication.
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, Shahram Izadi — ACM UIST, Oct 2016
Fusion4D: Real-time Performance Capture of Challenging Scenes
We contribute a new pipeline for live multi-view performance capture, generating temporally coherent high-quality reconstructions in real-time. Our algorithm supports both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion. Our approach is highly robust to both large frame-to-frame motion and topology changes, allowing us to reconstruct extremely challenging scenes. We demonstrate advantages over related real-time techniques that either deform an online generated template or continually fuse depth data nonrigidly into a single reference model. Finally, we show geometric reconstruction results on par with offline methods which require orders of magnitude more processing time and many more RGBD cameras.
Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, Shahram Izadi — ACM SIGGRAPH, July 2016
HyperDepth: Learning Depth from Structured Light Without Matching
Structured light sensors are popular due to their robustness to untextured scenes and multipath. These systems triangulate depth by solving a correspondence problem between each camera and projector pixel. This is often framed as a local stereo matching task, correlating patches of pixels in the observed and reference image. However, this is computationally intensive, leading to reduced depth accuracy and framerate. We contribute an algorithm for solving this correspondence problem efficiently, without compromising depth accuracy. For the first time, this problem is cast as a classification-regression task, which we solve extremely efficiently using an ensemble of cascaded random forests. Our algorithm scales in number of disparities, and each pixel can be processed independently, and in parallel. No matching or even access to the corresponding reference pattern is required at runtime, and regressed labels are directly mapped to depth. Our GPU-based algorithm runs at a 1KHz for 1.3 MP input/output images, with disparity error of 0.1 subpixels. We show a prototype high framerate depth camera running at 375Hz, useful for solving tracking-related problems. We demonstrate our algorithmic performance, creating high resolution real-time depth maps that surpass the quality of current state of the art depth technologies, highlighting quantization-free results with reduced holes, edge fattening and other stereo-based depth artifacts.
Sean Ryan Fanello, Christoph Rhemann, Vladimir Tankovich, Adarsh Kowdle, Sergio Orts Escolano, David Kim, Shahram Izadi — IEEE CVPR, June 2016 (Oral)
FlexCase: Enhancing Mobile Interaction with a Flexible Sensing and Display Cover
FlexCase is a novel flip cover for smartphones, which brings flexible input and output capabilities to existing mobile phones. It combines an e-paper display with a pressure- and bend-sensitive input sensor to augment the capabilities of a phone. Due to the form factor, FlexCase can be easily transformed into several different configurations, each with different interaction possibilities. Users can use FlexCase to perform a variety of touch, pressure, grip and bend gestures in a natural manner, much like interacting with a sheet of paper. The secondary e-paper display can act as a mechanism for providing user feedback and persisting content from the main display. In this paper, we explore the rich design space of FlexCase and present a number of different interaction techniques. Beyond, we highlight how touch and flex sensing can be combined to support a novel type of gestures, which we call Grip & Bend gestures. We also describe the underlying technology and gesture sensing algorithms. Numerous applications apply the interaction techniques in convincing real-world examples, including enhanced e-paper reading and interaction, a new copy and paste metaphor, high degree of freedom 3D and 2D manipulation, and the ability to transfer content and support input between displays in a natural and flexible manner.
Christian Rendl, David Kim, Patrick Parzer, Sean Fanello, Martin Zirkl, Gregor Scheipl, Michael Haller, Shahram Izadi — ACM CHI, San Jose, CA, USA, May 2016. (Best Paper Award)
Casalendar: Integrating the Smart Home into the Digital Calendar
With the growing adoption of smart home technologies, inhabitants are faced with the challenge of making sense of the data that their homes can collect to configure automated behaviors that benefit their routines. Current commercial smart home interfaces usually provide information on individual devices instead of a more comprehensive overview of a home’s behavior. To reduce the complexity of smart home data and integrate it better into inhabitants’ lives, we turned to the familiar metaphor of a calendar and developed our smart home interface Casalendar. In order to investigate the concept and evaluate our goals to facilitate the understanding of smart home data, we created a prototype that we installed in two commercial smart homes for a month. The results we present in this paper are based on our analysis of user data from questionnaires, semi-structured interviews, participant-driven audio and screenshot feedback as well as logged interactions with our system. Our findings exposed advantages and disadvantages of this metaphor, emerging usage patterns, privacy concerns and challenges for information visualization. We further report on implications for design and open challenges we revealed through this work.
Sarah Mennicken, David Kim, Elaine May Huang — ACM CHI, San Jose, CA, USA, May 2016
SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips
We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment whilst interactively segmenting the scene simply by reaching out and touching any desired object or surface. Our system continuously learns from these segmentations, and labels new unseen parts of the environment. Unlike offline systems where capture, labeling, and batch learning often take hours or even days to perform, our approach is fully online. This provides users with continuous live feedback of the recognition during capture, allowing to immediately correct errors in the segmentation and/or learning—a feature that has so far been unavailable to batch and offline methods. This leads to models that are tailored or personalized specifically to the user’s environments and object classes of interest, opening up the potential for new applications in augmented reality, interior design, and human/robot navigation. It also provides the ability to capture substantial labeled 3D datasets for training large-scale visual recognition systems.
Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip Torr — ACM TOG, August 2015
Accurate, Robust, and Flexible Real-time Hand Tracking
We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.
Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, Daniel Freedman, Pushmeet Kohli, Eyal Krupka, Andrew Fitzgibbon, Shahram Izadi — ACM CHI, Seoul, South Korea April 2015 (Honorable Mention)
FlexSense: A Transparent Self-Sensing Deformable Surface
We present FlexSense, a new thin-film, transparent sensing surface based on printed piezoelectric sensors, which can reconstruct complex deformations without the need for any external sensing, such as cameras. Done in collaboration with Media Interaction Lab, Hagenberg, Austria and the Institute of Surface Technologies and Photonics, Joanneum Research, FlexSense provides a fully self-contained setup which improves mobility and is not affected from occlusions. Using only a sparse set of sensors, printed on the periphery of the surface substrate, we devise two new algorithms to fully reconstruct the complex deformations of the sheet, using only these sparse sensor measurements. An evaluation shows that both proposed algorithms are capable of reconstructing complex deformations accurately. We demonstrate how FlexSense can be used for a variety of 2.5D interactions, including as a transparent cover for tablets where bending can be performed alongside touch to enable magic lens style effects, layered input, and mode switching, as well as the ability to use our device as a high degree-of-freedom input controller for gaming and beyond.
Christian Rendl, David Kim, Sean Fanello, Patrick Parzer, Christoph Rhemann, Jonathan Taylor, Martin Zirkl, Gregor Scheipl, Thomas Rothländer, Michael Haller, Shahram Izadi — ACM UIST, Honolulu, HI, USA, Oct 2014
The PumpSpark Fountain Development Kit
The PumpSpark Fountain Development Kit includes a controller, eight miniature water pumps, and various accessories to allow rapid prototyping of fluidic user interfaces. The controller provides both USB and logic-level serial interfaces, yielding fast (~ 100ms), high-resolution (8-bit) control of water streams up to about 1 meter high. Numerous example applications built using the PumpSpark kit are presented. The kit has been the subject of a student contest with over 100 students, demonstrating its utility in rapid prototyping of fluidic systems.
Paul H Dietz, Gabriel Reyes, David Kim — ACM DIS, Vancouver, BC, Canada, June 2014
MixFab: A Mixed-Reality Environment for Personal Fabrication
Personal fabrication machines, such as 3D printers and laser cutters, are becoming increasingly ubiquitous. However, designing objects for fabrication still requires 3D modeling skills, thereby rendering such technologies inaccessible to a wide user-group. In this paper, we introduce MixFab, a mixed-reality environment for personal fabrication that lowers the barrier for users to engage in personal fabrication. Users design objects in an immersive augmented reality environment, interact with virtual objects in a direct gestural manner and can introduce existing physical objects effortlessly into their designs. We describe the design and implementation of MixFab, a user-defined gesture study that informed this design, show artifacts designed with the system and describe a user study evaluating the system’s prototype.
Christian Weichel, Manfred Lau, David Kim, Nicolas Villar, Hans W Gellersen — ACM CHI, Toronto, ON, Canada, April 2014 (Best Paper Award)
RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces
We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom “physical widgets” fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.
David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Nießner, D Alex Butler, Sean Fanello, Vivek Pradeep — ACM CHI, Toronto, ON, Canada, April 2014 (Best Paper Award)
Digits: Freehand 3D Interactions Anywhere Using a Wrist-Worn Gloveless Sensor
Digits is a wrist-worn sensor that recovers the full 3D pose of the user’s hand. This enables a variety of freehand interactions on the move. The system targets mobile settings, and is specifically designed to be low-power and easily reproducible using only off-the-shelf hardware. The electronics are self-contained on the user’s wrist, but optically image the entirety of the user’s hand. This data is processed using a new pipeline that robustly samples key parts of the hand, such as the tips and lower regions of each finger. These sparse samples are fed into new kinematic models that leverage the biomechanical constraints of the hand to recover the 3D pose of the user’s hand. The proposed system works without the need for full instrumentation of the hand (for example using data gloves), additional sensors in the environment, or depth cameras which are currently prohibitive for mobile scenarios due to power and form-factor considerations. We demonstrate the utility of Digits for a variety of application scenarios, including 3D spatial interaction with mobile devices, eyes-free interaction on-the-move, and gaming. We conclude with a quantitative and qualitative evaluation of our system, and discussion of strengths, limitations and future work.
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, Patrick Olivier — ACM UIST, Cambridge, MA, USA, Oct 2012 (Best Demo Award 2nd place)
Interactive environment-aware handheld projectors for pervasive computing spaces
This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are “aware” of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions including direct touch, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities.
David Molyneaux, Shahram Izadi, David Kim, Otmar Hilliges, Steve Hodges, Xiang Cao, Alex Butler, Hans Gellersen — PERVASIVE, Springer, Newcastle upon Tyne, UK, Jun 2012 (Best Paper Award)
HoloDesk: Direct 3D Interactions with a Situated See-Through Display
HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display and to directly interact with an spatially aligned 3D virtual world, without the need for any specialized head-worn hardware or input device. We introduce a new technique for interpreting raw Kinect data to approximate and track rigid (e.g., books, cups) and non-rigid (e.g., hands, paper) physical objects and support a variety of physics-inspired interactions between virtual and real. In particular the algorithm models natural human grasping of virtual objects with more fidelity than previously demonstrated. A qualitative study highlights rich emergent 3D interactions, using hands and real-world objects. The implementation of HoloDesk is described in full, and example application scenarios explored. Finally, HoloDesk is quantitatively evaluated in a 3D target acquisition task, comparing the system with indirect and glasses-based variants.
Otmar Hilliges, David Kim, Shahram Izadi, Malte Weiss, Andrew Wilson — ACM CHI, Austin, TX, USA, May 2012
KinectFusion: Real-Time 3D Reconstruction and Interaction using a Moving Depth Camera
KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, Andrew Fitzgibbon — ACM UIST, Santa Barbara, CA, USA, Oct 2011
(KinectFusion) Real-Time Dense Surface Mapping and Tracking with Kinect
We present a system for accurate real-time mapping of complex and arbitrary indoor scenes in variable lighting conditions, using only a moving low-cost depth camera and commodity graphics hardware. We fuse all of the depth data streamed from a Kinect sensor into a single global implicit surface model of the observed scene in real-time. The current sensor pose is simultaneously obtained by tracking the live depth frame relative to the global model using a coarse-to-fine iterative closest point (ICP) algorithm, which uses all of the observed depth data available. We demonstrate the advantages of tracking against the growing full surface model compared with frame-to-frame tracking, obtaining tracking and mapping results in constant time within room sized scenes with limited drift and high accuracy. We also show both qualitative and quantitative results relating to various aspects of our tracking and mapping system. Modelling of natural scenes, in real-time with only commodity sensor and GPU hardware, promises an exciting step forward in augmented reality (AR), in particular, it allows dense surfaces to be reconstructed in real-time, with a level of detail and robustness beyond any solution yet presented using passive computer vision.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, Andrew Fitzgibbon — IEEE ISMAR. Basel, Switzerland, Oct 2011 (Best Paper Award)
Vermeer: Direct Interaction with a 360° Viewable 3D Display
We present Vermeer, a novel interactive 360° viewable 3D display. Like prior systems in this area, Vermeer provides viewpoint-corrected, stereoscopic 3D graphics to simultaneous users, 360° around the display, without the need for eyewear or other user instrumentation. Our goal is to over-come an issue inherent in these prior systems which – typically due to moving parts – restrict interactions to outside the display volume. Our system leverages a known optical illusion to demonstrate, for the first time, how users can reach into and directly touch 3D objects inside the display volume. Vermeer is intended to be a new enabling technology for interaction, and we therefore describe our hardware implementation in full, focusing on the challenges of combining this optical configuration with an existing approach for creating a 360° viewable 3D display. Initially we demonstrate direct involume interaction by sensing user input with a Kinect camera placed above the display. However, by exploiting the properties of the optical configuration, we also demonstrate novel prototypes for fully integrated input sensing alongside simultaneous display. We conclude by discussing limitations, implications for interaction, and ideas for future work.
Alex Butler, Otmar Hilliges, Shahram Izadi, Steve Hodges, David Molyneaux, David Kim, Danny Kong — ACM UIST. Santa Barbara, CA, USA. Oct 2011
Shake'n'Sense: Reducing Interference for Overlapping Structured Light Depth Cameras
We present a novel yet simple technique that mitigates the interference caused when multiple structured light depth cameras point at the same part of a scene. The technique is particularly useful for Kinect, where the structured light source is not modulated. Our technique requires only mechanical augmentation of the Kinect, without any need to modify the internal electronics, firmware or associated host software. It is therefore simple to replicate. We show qualitative and quantitative results highlighting the improvements made to interfering Kinect depth signals. The camera frame rate is not compromised, which is a problem in approaches that modulate the structured light source. Our technique is non-destructive and does not impact depth values or geometry. We discuss uses for our technique, in particular within instrumented rooms that require simultaneous use of multiple overlapping fixed Kinect cameras to support whole room interactions.
D Alex Butler, Shahram Izadi, Otmar Hilliges, David Molyneaux, Steve Hodges, David Kim — ACM CHI. Austin, TX, USA. May 2012 (Honorable Mention)
Multi-Touch Authentication on Tabletops
The introduction of tabletop interfaces has given rise to the need for the development of secure and usable authentication techniques that are appropriate for the co-located collaborative settings for which they have been designed. Most commonly, user authentication is based on something you know, but this is a particular problem for tabletop interfaces, as they are particularly vulnerable to shoulder surfing given their remit to foster co-located collaboration. In other words, tabletop users would typically authenticate in full view of a number of observers. In this paper, we introduce and evaluate a number of novel tabletop authentication schemes that exploit the features of multi-touch interaction in order to inhibit shoulder surfing. In our pilot work with users, and in our formal user-evaluation, one authentication scheme – Pressure-Grid – stood out, significantly enhancing shoulder surfing resistance when participants used it to enter both PINs and graphical passwords.
David Kim, Paul Dunphy, Pam Briggs, Jonathan Hook, John W Nicholson, James Nicholson, Patrick Olivier — ACM CHI. Atlanta, Georgia, USA. Apr 2010
Creating Malleable Interactive Surfaces using Liquid Displacement Sensing
We present a new approach for rapidly prototyping multi-touch and object sensing surfaces. It works by liquid displacement inside a malleable projection surface. The system provides both touch and pressure information and a distinct organic quality when touched. The system is easy to build and produces a clean signal revealing multiple fingers, whole hands and other object outlines that can be processed using computer vision techniques. This approach provides an easy mechanism to build interactive surfaces, requiring no infrared edge lighting or soldering. In this paper we provide an overview of the approach, some of its unique capabilities, and uncover some of the tradeoffs between viscosity of liquid, air pressure, surface malleability and the volume of liquid used. Our aim is to allow practitioners – from DIY enthusiasts to researchers – to build and experiment with such systems more readily.
Otmar Hilliges, David Kim, Shahram Izadi — IEEE Tabletop and Interactive Surfaces. Amsterdam, Netherlands. Oct 2008
Designing for Collaborative Creative Problem Solving
Collaborative creativity is traditionally supported by formal techniques, such as brainstorming. These techniques improve the idea-generation process by creating group synergies, but also suffer from a number of negative effects. Current electronic tools to support collaborative creativity overcome some of these problems, but introduce new ones, by either losing the benefits of face-to-face communication or the immediacy of simultaneous contribution. Using an interactive environment as a test bed, we are investigating how collaborative creativity can be supported electronically while maintaining face-to-face communication. What are the design-factors influencing such a system? We have designed a brainstorming application that uses an interactive table and a large wall display, and compared the results of using it to traditional paper-based brainstorming in a user study with 30 participants. From the considerations that went into the design and the observations during the study we derive a number of design guidelines for collaborative systems in interactive environments.
Otmar Hilliges, Lucia Terrenghi, Sebastian Boring, David Kim, Hendrik Richter, Andreas Butz — ACM Creativity & Cognition. Washington D.C., USA. Jun 2007
Granted Patents
Stereo image processing using contours (US 9269018 B2). David Kim, Shahram Izadi, Christoph Rhemann, Christopher Zach. 05 May 2016
Generating computer models of 3d objects (US 9053571 B2). Jamie Daniel Joseph Shotton, Shahram Izadi, Otmar Hilliges, David Kim, David Molyneaux, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges. 9 Jun 2015.
Moving object segmentation using depth images (US 8401225 B2). Richard Newcombe, Shahram Izadi, Otmar Hilliges, David Kim, David Molyneaux, Jamie Daniel Joseph Shotton, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges, David Alexander Butler.
Real-time camera tracking using depth maps (US8401242 B2). Richard Newcombe, Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges, David Alexander Butler.
Gesture Recognition Techniques (US 20120306734 A1). David Kim, Otmar D Hilliges, Shahram Izadi, Patrick L Olivier, Jamie Daniel Joseph Shotton, Pushmeet Kohli, David G Molyneaux, Stephen E Hodges, Andrew W Fitzgibbon.
Mobile Camera Localization Using Depth Maps (US 20120194644 A1). Richard Newcombe, Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges, David Alexander Butler.
Using a three-dimensional environment model in gameplay (US 8570320 B2). Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Pushmeet Kohli, Andrew Fitzgibbon, Stephen Edward Hodges, David Alexander Butler.
Distributed asynchronous localization and mapping for augmented reality (US 8933931 B2). Alexandru Balan, Jason Flaks, Steve Hodges, Michael Isard, Oliver Williams, Paul Barham, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim. 13 Jan 2015
Three-dimensional environment reconstruction (US 8587583 B2). Richard Newcombe, Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Stephen Edward Hodges, David Alexander Butler, Andrew Fitzgibbon, Pushmeet Kohli.
Human Body Pose Estimation (US 20110210915 A1). Jamie Daniel Joseph Shotton, Shahram Izadi, Otmar Hilliges, David Kim, David Geoffrey Molyneaux, Matthew Darius Cook, Pushmeet Kohli, Antonio Criminisi, Ross Brook Girshick, Andrew William Fitzgibbon.
Learning Image Processing Tasks from Scene Reconstructions (US8971612 B2). Jamie Daniel Joseph Shotton, Pushmeet Kohli, Stefan Johannes Josef Holzer, Shahram Izadi, Carsten Curt Eckard Rother, Sebastian Nowozin, David Kim, David Molyneaux, Otmar Hilliges. 03 Mar 2015.
Reducing interference between multiple infra-red depth cameras (US9247238 B2). Shahram Izadi, David Molyneaux, Otmar Hilliges, David Kim, Jamie Daniel Joseph Shotton, Stephen Edward Hodges, David Alexander Butler, Andrew Fitzgibbon, Pushmeet Kohli. 26 Jan 2016.
Other Co-Authored Patent Applications
User interaction in augmented reality (US20120113223 A1). Otmar Hilliges, David Kim, Shahram Izadi, David Molyneaux, Stephen Edward Hodges, David Alexander Butler. 04 Nov 2010
Augmented reality with direct user interaction (US20120113140 A1). Otmar Hilliges, David Kim, Shahram Izadi, David Molyneaux, Stephen Edward Hodges, David Alexander Butler. 04 Nov 2010
Wearable sensor for tracking articulated body-parts (US20140098018 A1). David Kim, Shahram Izadi, Otmar Hilliges, David Alexander Butler, Stephen Hodges, Patrick Luke Olivier, Jiawen Chen, Iason Oikonomidis. 04 Oct 2012
Grasping virtual objects in augmented reality (US20140104274 A1). Otmar Hilliges, David Kim, Shahram Izadi, Malte Hanno Weiss. 17 Oct 2012
Authentication by multi-level pressure exertion on multi-touch tabletop interfaces (GB 2476822 A). Paul Dunphy, David Kim, James Nicholson, John Nicholson, Jonathan Hook Pamela Briggs, Patrick Olivier
Depth sensing using an infrared camera (US 20150248764 A1). Cem Keskin, Sean Ryan Francesco Fanello, Shahram Izadi, Pushmeet Kohli, David Kim, David Sweeney, Jamie Daniel Joesph Shotton, Duncan Paul ROBERTSON, Sing Bing Kang. 03 Sep 2015
Contour-based classification of objects (US 20150199592 A1). David Kim, Cem Keskin, Jamie Daniel Joseph Shotton, Shahram Izadi. 16 Jul 2015.
3d silhouette sensing system (US 20150199018 A1). David Kim, Shahram Izadi, Vivek Pradeep, Steven Bathiche, Timothy Andrew Large, Karlton David Powell. 16 Jul 2015.
Using photometric stereo for 3d environment modeling (US 20140184749 A1). Otmar Hilliges, Malte Hanno Weiss, Shahram Izadi, David Kim, Carsten Curt Eckard Rother. 28 Dec 2012