|
2016
|
|
Roto++: Accelerating Professional Rotoscoping using Shape Manifolds
W. Li, F. Viola, J. Starck, G. J. Brostow, N. D.F. Campbell
Rotoscoping (cutting out different characters/objects/layers in raw video footages) is a ubiquitous task in modern post-production and represents a significant investment in person-hours. In this work, we study the particular task of professional rotoscoping for high-end, live action movies and propose a new framework that works with roto-artists to accelerate the workflow and improve their productivity.
Working with the existing keyframing paradigm, our first contribution is the development of a shape model that is updated as artists add successive keyframes. This model is used to improve the output of traditional interpolation and tracking techniques, reducing the number of keyframes that need to be specified by the artist. Our second contribution is to use the same shape model to provide a new interactive tool that allows an artist to reduce the time spent editing each keyframe. The more keyframes that are edited, the better the interactive tool becomes, accelerating the process and making the artist more efficient without compromising their control. Finally, we also provide a new, professionally rotoscoped dataset that enables truly representative, real-world evaluation of rotoscoping methods. We used this dataset to perform a number of experiments, including an expert study with professional roto-artists, to show, quantitatively,the advantages of our approach.Read more...Show less
|
|
|
|
Structured Prediction of Unobserved Voxels From a Single Depth Image
M. Firman, O. Mac Aodha, S. Julier, G. J. Brostow
Building a complete 3D model of a scene, given only a single depth image, is underconstrained. To gain a full volumetric model, one needs either multiple views, or a single view together with a library of unambiguous 3D models that will fit the shape of each individual object in the scene.
We hypothesize that objects of dissimilar semantic classes often share similar 3D shape components, enabling a limited dataset to model the shape of a wide range of objects, and hence estimate their hidden geometry. Exploring this hypothesis, we propose an algorithm that can complete the unobserved geometry of tabletop-sized objects, based on a supervised model trained on already available volumetric elements. Our model maps from a local observation in a single depth image to an estimate of the surface shape in the surrounding neighborhood. We validate our approach both qualitatively and quantitatively on a range of indoor object collections and real challenging scenes. Read more...Show less
|
|
|
|
Scalable Inside-Out Image-Based Rendering
P. Hedman, T. Ritschel, G. Drettakis, G. Brostow
Our aim is to give users real-time free-viewpoint rendering of real
indoor scenes, captured with off-the-shelf equipment such as a high-quality color camera and a commodity depth sensor.
Image-based Rendering (IBR) can provide the realistic imagery required at realtime speed. For indoor scenes however, two challenges are especially prominent. First, the reconstructed 3D geometry must be compact, but faithful enough to respect occlusion relationships when viewed up close. Second, man-made materials call for view-dependent texturing, but using too many input photographs reduces performance. We customize a typical RGB-D 3D surface reconstruction pipeline to produce a coarse global 3D surface, and local, per-view geometry for each input image. Our tiled IBR preserves quality by economizing on the expected contributions that entire groups of input pixels make to a final image. The two components are designed to work together, giving real-time performance, while hardly sacrificing quality. Testing on a variety of challenging scenes shows that our inside-out IBR scales favorably with the number of input images. Read more...Show less
|
|
|
|
Rapid, Detail-Preserving Image Downscaling
N. Weber, M. Waechter, S. C. Amend, S. Guthe and M. Goesele
Image downscaling is arguably the most frequently used image processing tool. We present an algorithm based on convolutional filters where input pixels contribute more to the output image the more their color deviates from their local neighborhood, which preserves visually important details. In a user study we verify that users prefer our results over related work. Our efficient GPU implementation works in real-time when downscaling images from 24 M to 70 k pixels. Further, we demonstrate empirically that our method can be successfully applied to videos. Read more...Show less
|
|
|
|
Shading-aware Multi-view Stereo
F. Langguth, K. Sunkavalli, S. Hadap, M. Goesele
We present a novel multi-view reconstruction approach that
effectively combines stereo and shape-from-shading energies into a single optimization scheme.
Our method uses image gradients to transition between
stereo-matching (which is more accurate at large gradients) and Lambertian shape-from-shading (which is more robust in flat regions). In addition, we show that our formulation is invariant to spatially varying albedo without explicitly modeling it. We show that the resulting energy function can be optimized efficiently using a smooth surface representation based on bicubic patches, and demonstrate that this algorithm outperforms both previous multi-view stereo algorithms and shading based refinement approaches on a number of datasets. Read more...Show less
|
|
|
|
Ghosting and Popping Detection for Image-Based Rendering
S. Guthe, D. Cunningham, P..Schardt, M. Goesele
Film sequences generated using image-based rendering techniques are commonly used in broadcasting, especially for sporting events. In many cases, however, image-based rending sequences contain artifacts, and these must be manually located.
Here, we propose an algorithm to automatically detect not only the presence of the two most disturbing classes of artifact (popping and ghosting), but also the strength of each instance of an artifact. A simple perceptual evaluation of the technique shows that it performs well. Read more...Show less
|
|
|
|
Automatic 3D Car Model Alignment for Mixed Image-Based Rendering
R. Ortiz-Cayon, A. Djelouah, F. Massa, M. Aubry, G. Drettakis
Image-Based Rendering (IBR) allows good-quality freeviewpoint
navigation in urban scenes, but suffers from artifacts on poorly reconstructed objects, e.g., reflective surfaces such as cars. To alleviate this problem, we propose a method that automatically identifies stock 3D models, aligns them in the 3D scene and performs morphing to better capture image contours. We do this by first adapting learning-based methods to detect and identify an object class and pose in images. We then propose a method which exploits all available information, namely partial and inaccurate 3D reconstruction, multi-view calibration, image contours and the 3D model to achieve accurate object alignment suitable for subsequent morphing. These steps provide models which are well-aligned in 3D and to contours in all the images of the multi-view dataset, allowing us to use the resulting model in our mixed IBR algorithm. Our results show significant improvement in image quality for free-viewpoint IBR, especially when moving far from the captured viewpoints. Read more...Show less
|
|
|
|
Multi-View Inpainting for Image-Based Scene Editing and Rendering
T. Thonat, E. Shechtman, S. Paris, G. Drettakis
We propose a method to remove objects such as people and cars from multi-view urban image datasets, enabling free-viewpoint Image-Based Rendering (IBR) in the edited scenes. Our method combines information from multi-view 3D reconstruction with image inpainting techniques, by formulating the problem as an optimization of a global patch-based objective function. We use IBR techniques to reproject information from neighboring views, and 3D multi-view stereo reconstruction to perform multi-view coherent initialization for inpainting of pixels not filled by reprojection. Our algorithm performs multi-view consistent inpainting for color and 3D by blending reprojections with patch-based image inpainting. We run our algorithm on casually captured datasets, and Google Street View data, removing objects such as cars, people and pillars, showing that our approach produces results of sufficient quality for free-viewpoint IBR on “cleaned up” scenes, as well as IBR scene editing, such as limited displacement of real objects Read more...Show less
|
|
|
|
Cotemporal Multi-View Video Segmentation
A. Djelouah, J.S. Franco, E. Boyer, P. Pérez, G. Drettakis
We address the problem of multi-view video segmentation of dynamic scenes in general and outdoor environments with possibly moving cameras. Multi-view methods for dynamic scenes usually rely on geometric calibration to impose spatial shape constraints between viewpoints. In this paper, we show that the calibration constraint can be relaxed while still getting competitive segmentation results using multi-view constraints. We introduce new multi-view cotemporality constraints through motion correlation cues, in addition to common appearance features used by cosegmentation methods to identify co-instances of objects. We also take advantage of learning based segmentation strategies by casting the problem as the selection of monocular proposals that satisfy multi-view constraints. This yields a fully automated method that can segment subjects of interest without any particular pre-processing stage. Results on several challenging outdoor datasets demonstrate the feasibility and robustness of our approach Read more...Show less
|
|
|
|
Evaluation of a new asset creation pipeline for indie game developers
C. Fidas, D. Halvatzaras, N. Avouris, I. Orvieto
Indie game developers need to produce high quality games with constraint resources in order to survive in the highly competitive game industry today. Indicative of the high competition is that, video game development cycles increased for today’s high quality video games, resulting in considerable increase in budgets. Introduction of new tools and methods of work is welcome as this is an industry characterized by high degree of technological innovation, however involvement in evaluation of new tools and methods in this context may create a major disruption given the dire resource constraints of the developers. Given this context, an evaluation study of new tools in the indie game industry needs to be conducted with caution, in order to be successful and produce valid results. This paper outlines the design of such evaluation study and the main considerations involved. Read more...Show less
|
|
|
2015
|
Multi-View Intrinsic Images of Outdoors Scenes with an Application to Relighting
Sylvain Duchêne, Clement Riant, Gaurav Chaurasia, Jorge Lopez-Moreno, Pierre-Yves Laffont, Stefan Popov, Adrien Bousseau, George Drettakis
With the proliferation of acquisition devices, gathering massive volumes of 3D data is now easy. Processing such large masses of pointclouds, however, remains a challenge. This is particularly a problem for raw scans with missing data, noise, and varying sampling density.
In this work, we present a simple, scalable, yet powerful data reconstruction algorithm. We focus on reconstruction of man-made scenes as regular arrangements of planes (RAP), thereby selecting both local plane-based approximations along with their global inter-plane relations. We propose a novel selection formulation to directly balance between data fitting and the simplicity of the resulting arrangement of extracted planes. The main technical contribution is a formulation that allows less-dominant orientations to still retain their internal regularity, and not become overwhelmed and regularized by the dominant scene orientations. We evaluate our approach on a variety of complex 2D and 3D pointclouds, and demonstrate the advantages over existing alternative methods.Read more...Show less
|
|
|
|
RAPter: Rebuilding Man-made Scenes with Regular Arrangements of Planes
Aron Monszpart, Nicolas Mellado, Gabriel J. Brostow, Niloy J. Mitra
With the proliferation of acquisition devices, gathering massive volumes of 3D data is now easy. Processing such large masses of pointclouds, however, remains a challenge. This is particularly a problem for raw scans with missing data, noise, and varying sampling density. In this work, we present a simple, scalable, yet powerful data reconstruction algorithm. We focus on reconstruction of man-made scenes as regular arrangements of planes (RAP), thereby selecting both local plane-based approximations along with their global inter-plane relations. We propose a novel selection formulation to directly balance between data fitting and the simplicity of the resulting arrangement of extracted planes. The main technical contribution is a formulation that allows less-dominant orientations to still retain their internal regularity, and not become overwhelmed and regularized by the dominant scene orientations. We evaluate our approach on a variety of complex 2D and 3D pointclouds, and demonstrate the advantages over existing alternative methods.Read more...Show less
|
|
|
|
Learning to Remove Soft Shadows
Maciej Grika, Michael Terry and Gabriel J. Brostow
Manipulated images lose believability if the user’s edits fail to account for shadows. We propose a method that makes removal and editing of soft shadows easy. Soft shadows are ubiquitous, but remain notoriously difficult to extract and manipulate.
We posit that soft shadows can be segmented, and therefore edited, by learning a mapping function for image patches that generates shadow mattes. We validate this premise by removing soft shadows from photographs with only a small amount of user input. Given only broad user brush strokes that indicate the region to be processed, our new supervised regression algorithm automatically unshadows an image, removing the umbra and penumbra. The resulting lit image is frequently perceived as a believable shadow-free version of the scene. We tested the approach on a large set of soft shadow images, and performed a user study that compared our method to the state of the art and to real lit scenes. Our results are more difficult to identify as being altered, and are perceived as preferable compared to prior work. Read more...Show less
|
|
|
|
Requirements Elicitation for new Video Game Development Tools: A Case Study
Christos Fidas, Nikolaos Avouris, Ivan Orvieto
This paper presents a case study involving requirements elicitation for new tools in video games development. Eight video game developer companies from three different countries and a variety of stakeholders (N=17) participated in a user study that was based on a tailor-made requirements elicitation framework.
During the process, interesting issues emerged related to the applied method but as well concerning the presentation of innovative tools, as disruptive technology, the different stakeholders’ points of view and roles in the process, the role of technology providers and the organizational challenges towards this new game development pipeline. This case study provides interesting insights in applying a user-centered approach for requirements elicitation in the video game application domain and discusses lessons learned which can be of value for UX researchers and practitioners in video game research.Read more...Show less
|
|
|
|
A Bayesian Approach for Selective Image-Based Rendering
Rodrigo Ortiz-Cayon, Abdelaziz Djelouah, George Drettakis
Image-Based Rendering (IBR) algorithms generate high quality photo-realistic imagery without the burden of de-tailed modeling and expensive realistic rendering. Recent methods have different strengths and weaknesses, depending on 3D reconstruction quality and scene content.
Each algorithm operates with a set of hypotheses about the scene and the novel views, resulting in different quality/speed trade-offs in different image regions. We present a principled approach to select the algorithm with the best quality/speed trade-off in each region. To do this, we propose a Bayesian approach, modeling the rendering quality, the rendering process and the validity of the assumptions of each algorithm. We then choose the algorithm to use withMaximum a Posteriori estimation. We demonstrate the utility of our approach on recent IBR algorithms which useover segmentation and are based on planar reprojection and shape-preserving warps respectively. Our algorithm selects the best rendering algorithm for each superpixel in a preprocessing step; at runtime our selective IBR uses this choice to achieve significant speed up at equivalent or better quality compared to previous algorithms.Read more...Show less
|
|
|
|
Accurate Isosurface Interpolation with Hermite Data
Simon Fuhrmann, Misha Kazhdan and Michael Goesele
In this work we study the interpolation problem in contouring methods such as Marching Cubes. Traditionally, linear interpolation is used to define the position of an isovertex along a zero-crossing edge, which is a suitable approach if the underlying implicit function is (approximately) piecewise linear along each edge.
Non-linear implicit functions, however, are frequently encountered and linear interpolation leads to inaccurate isosurfaces with visible reconstruction artifacts. We instead utilize the gradient of the implicit function to generate more accurate isosurfaces by means of Hermite interpolation techniques. We propose and compare several interpolation methods and demonstrate clear quality improvements by using higher order interpolants. We further show the effectiveness of the approach even when Hermite data is not available and gradients are approximated using finite differences.Read more...Show less
|
|
|
|
Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled
Clement Godard, Peter Hedman, Wenbin Li, Gabriel Brostow
Reconstructing the surface of highly specular objects is a challenging task. The shapes of diffuse and rough specular objects can be captured in an uncontrolled setting using consumer equipment.
In contrast, highly specular objects have previously deterred capture in uncontrolled environments and have only been reconstructed using tailor-made hardware. We propose a method to reconstruct such objects in uncontrolled environments using only commodity hardware. As input, our method expects multi-view photographs of the specular object, its silhouettes and an environment map of its surroundings. We compare the reflected colors in the photographs with the ones in the environment to form probability distributions over the surface normals. As the effect of inter-reflections cannot be ignored for highly specular objects, we explicitly model them when forming the probability distributions. We recover the shape of the object in an iterative process where we alternate between estimating normals and updating the shape of the object to better explain these normals. We run experiments on both synthetic and real-world data, that show our method is robust and produces accurate reconstructions with as few as 25 input photographs.Read more...Show less
|
|
|
|
MVE—An image-based reconstruction environment
Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, Michael Goesele
In this work we study the interpolation problem in contouring methods such as Marching Cubes. Traditionally, linear interpolation is used to define the position of an isovertex along a zero-crossing edge, which is a suitable approach if the underlying implicit function is (approximately) piecewise linear along each edge.
Non-linear implicit functions, however, are frequently encountered and linear interpolation leads to inaccurate isosurfaces with visible reconstruction artifacts. We instead utilize the gradient of the implicit function to generate more accurate isosurfaces by means of Hermite interpolation techniques. We propose and compare several interpolation methods and demonstrate clear quality improvements by using higher order interpolants. We further show the effectiveness of the approach even when Hermite data is not available and gradients are approximated using finite differences.Read more...Show less
|
|
|
|
A Survey of Photometric Stereo Techniques
Jens Ackermann and Michael Goesele
Reconstructing the shape of an object from images is an important problem in computer vision that has led to a variety of solution strategies. This survey covers photometric stereo, i.e., techniques that exploit the observed intensity variations caused by illumination changes to recover the orientation of the surface.
In the most basic setting, a diffuse surface is illuminated from at least three directions and captured with a static camera. Under some conditions, this allows to recover per-pixel surface normals. Modern approaches generalize photometric stereo in various ways, e.g., relaxing constraints on lighting, surface reflectance and camera placement or creating different types of local surface estimates. Starting with an introduction for readers unfamiliar with the subject, we discuss the foundations of this field of research. We then summarize important trends and developments that emerged in the last three decades. We put a focus on approaches with the potential to be applied in a broad range of scenarios. This implies, e.g., simple capture setups, relaxed model assumptions, and increased robustness requirements. The goal of this review is to provide an overview of the diverse concepts and ideas on the way towards more general techniques than traditional photometric stereo.Read more...Show less
|
|
|
2014
|
|
C-LOD: Context-aware Material Level-Of-Detail applied to Mobile Graphics
George Alex Koulieris, George Drettakis, Douglas Cunningham, Katerina Mania
Attention-based Level-Of-Detail (LOD) managers downgrade the quality of areas that are expected to go unnoticed by an observer to economize on computational resources. The perceptibility of lowered visual fidelity is determined by the accuracy of the attention model that assigns quality levels.
Most previous attention based LOD managers do not take into account saliency provoked by context, failing to provide consistently accurate attention predictions. This work extends a recent high level saliency model with four additional components yielding more accurate predictions: an object-intrinsic factor accounting for canonical form of objects, an object-context factor for contextual isolation of objects, a feature uniqueness term that accounts for the number of salient features in an image, and a temporal context that generates recurring fixations for objects inconsistent with the context. A perceptual experiment has been conducted to acquire the weighting factors to initialize our model. C-LOD has been designed, a LOD manager that maintains a constant frame rate on mobile devices by dynamically re-adjusting material quality on secondary visual features of non-attended objects. A proof of concept study has demonstrated that by incorporating C-LOD, complex effects such as parallax occlusion mapping usually omitted in mobile devices can now be employed, without overloading GPU capability and, at the same time, conserving battery power.Read more...Show less
|
|
|
|
An Automated High Level Saliency Predictor for Smart Game Balancing
George Alex Koulieris, George Drettakis, Douglas Cunningham, Katerina Mania
Successfully predicting visual attention can significantly improve many aspects of computer graphics: scene design, interactivity and rendering. Most previous attention models are mainly based on low-level image features, and fail to take into account high-level factors such as scene context, topology or task.
Low-level saliency has previously been combined with task maps, but only for predetermined tasks. Thus the application of these methods to graphics – e.g. for selective rendering – has not achieved its full potential. In this paper we present the first automated high level saliency predictor incorporating two hypotheses from perception and cognitive science which can be adapted to different tasks. The first states that a scene is comprised of objects expected to be found in a specific context as well objects out of context which are salient (scene schemata) while the other claims that viewer’s attention is captured by isolated objects (singletons). We propose a new model of attention by extending Eckstein’s Differential Weighting Model. We conducted a formal eye-tracking experiment which confirmed that object saliency guides attention to specific objects in a game scene and determined appropriate parameters for a model. We present a GPU based system architecture that estimates the probabilities of objects to be attended in real-time. We embedded this tool in a game level editor to automatically adjust game level difficulty based on object saliency, offering a novel way to facilitate game design. We perform a study confirming that game level completion time depends on object topology as predicted by our sys. Read more...Show less
|
|
|
|
MVE – A Multi-View Reconstruction Environment
Simon Fuhrmann, Fabian Langguth, Michael Goesele
This work presents MVE, the Multi-View Environment. MVE is an end-to-end multi-view geometry reconstruction soft-ware which takes photos of a scene as input and produces a surface triangle mesh as result.
The system covers a structure-from-motion algorithm, multi-view stereo reconstruction, generation of extremely dense point clouds, and reconstruction of surfaces from point clouds. In contrast to most image-based geometry reconstruction approaches, the system is focused on reconstruction of multi-scale scenes, an important aspect in many areas such as cultural heritage. It allows to reconstruct large datasets containing some detailed regions with much higher resolution than the rest of the scene. The system provides a graphical user interface for structure-from-motion reconstruction, visual inspection of images, depth maps, and rendering of scenes and meshes. Read more...Show less
|
|
|
|
Floating Scale Surface Reconstruction
Simon Fuhrmann, Michael Goesele
Any sampled point acquired from a real-world geometric object or scene represents a finite surface area and not just a single surface point. Samples therefore have an inherent scale, very valuable information that has been crucial for high quality reconstructions.
This work introduces a new method for surface reconstruction from oriented, scale-enabled sample points which operates on large, redundant and potentially noisy point sets. The approach draws upon a simple yet efficient mathematical formulation to construct an implicit function as the sum of compactly supported basis functions. The implicit function has spatially continuous “floating” scale and can be readily evaluated without any preprocessing. The final surface is extracted as the zero-level set of the implicit function. One of the key properties of the approach is that it is virtually parameter-free even for complex, mixed-scale datasets. In addition, this method is easy to implement, scalable and does not require any global operations. This method has been evaluated on a wide range of datasets for which it compares favorably to popular classic and current methods.Read more...Show less
|
|
