-
N-BVH: Neural ray queries with bounding volume hierarchies
Authors:
Philippe Weier,
Alexander Rath,
Élie Michel,
Iliyan Georgiev,
Philipp Slusallek,
Tamy Boubekeur
Abstract:
Neural representations have shown spectacular ability to compress complex signals in a fraction of the raw data size. In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures, making them ideal candidates for neural compression. Here, the main challenge lies in finding good trade-offs between efficient compression and cheap inference while minimizing training tim…
▽ More
Neural representations have shown spectacular ability to compress complex signals in a fraction of the raw data size. In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures, making them ideal candidates for neural compression. Here, the main challenge lies in finding good trade-offs between efficient compression and cheap inference while minimizing training time. In the context of rendering, we adopt a ray-centric approach to this problem and devise N-BVH, a neural compression architecture designed to answer arbitrary ray queries in 3D. Our compact model is learned from the input geometry and substituted for it whenever a ray intersection is queried by a path-tracing engine. While prior neural compression methods have focused on point queries, ours proposes neural ray queries that integrate seamlessly into standard ray-tracing pipelines. At the core of our method, we employ an adaptive BVH-driven probing scheme to optimize the parameters of a multi-resolution hash grid, focusing its neural capacity on the sparse 3D occupancy swept by the original surfaces. As a result, our N-BVH can serve accurate ray queries from a representation that is more than an order of magnitude more compact, providing faithful approximations of visibility, depth, and appearance attributes. The flexibility of our method allows us to combine and overlap neural and non-neural entities within the same 3D scene and extends to appearance level of detail.
△ Less
Submitted 25 May, 2024;
originally announced May 2024.
-
ControlMat: A Controlled Generative Approach to Material Capture
Authors:
Giuseppe Vecchio,
Rosalie Martin,
Arthur Roullier,
Adrien Kaiser,
Romain Rouffet,
Valentin Deschaintre,
Tamy Boubekeur
Abstract:
Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, til…
▽ More
Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.
△ Less
Submitted 25 September, 2023; v1 submitted 4 September, 2023;
originally announced September 2023.
-
The Visual Language of Fabrics
Authors:
Valentin Deschaintre,
Julia Guerrero-Viu,
Diego Gutierrez,
Tamy Boubekeur,
Belen Masia
Abstract:
We introduce text2fabric, a novel dataset that links free-text descriptions to various fabric materials. The dataset comprises 15,000 natural language descriptions associated to 3,000 corresponding images of fabric materials. Traditionally, material descriptions come in the form of tags/keywords, which limits their expressivity, induces pre-existing knowledge of the appropriate vocabulary, and ult…
▽ More
We introduce text2fabric, a novel dataset that links free-text descriptions to various fabric materials. The dataset comprises 15,000 natural language descriptions associated to 3,000 corresponding images of fabric materials. Traditionally, material descriptions come in the form of tags/keywords, which limits their expressivity, induces pre-existing knowledge of the appropriate vocabulary, and ultimately leads to a chopped description system. Therefore, we study the use of free-text as a more appropriate way to describe material appearance, taking the use case of fabrics as a common item that non-experts may often deal with. Based on the analysis of the dataset, we identify a compact lexicon, set of attributes and key structure that emerge from the descriptions. This allows us to accurately understand how people describe fabrics and draw directions for generalization to other types of materials. We also show that our dataset enables specializing large vision-language models such as CLIP, creating a meaningful latent space for fabric appearance, and significantly improving applications such as fine-grained material retrieval and automatic captioning.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
MatFormer: A Generative Model for Procedural Materials
Authors:
Paul Guerrero,
Miloš Hašan,
Kalyan Sunkavalli,
Radomír Měch,
Tamy Boubekeur,
Niloy J. Mitra
Abstract:
Procedural material graphs are a compact, parameteric, and resolution-independent representation that are a popular choice for material authoring. However, designing procedural materials requires significant expertise and publicly accessible libraries contain only a few thousand such graphs. We present MatFormer, a generative model that can produce a diverse set of high-quality procedural material…
▽ More
Procedural material graphs are a compact, parameteric, and resolution-independent representation that are a popular choice for material authoring. However, designing procedural materials requires significant expertise and publicly accessible libraries contain only a few thousand such graphs. We present MatFormer, a generative model that can produce a diverse set of high-quality procedural materials with complex spatial patterns and appearance. While procedural materials can be modeled as directed (operation) graphs, they contain arbitrary numbers of heterogeneous nodes with unstructured, often long-range node connections, and functional constraints on node parameters and connections. MatFormer addresses these challenges with a multi-stage transformer-based model that sequentially generates nodes, node parameters, and edges, while ensuring the semantic validity of the graph. In addition to generation, MatFormer can be used for the auto-completion and exploration of partial material graphs. We qualitatively and quantitatively demonstrate that our method outperforms alternative approaches, in both generated graph and material quality.
△ Less
Submitted 15 August, 2022; v1 submitted 3 July, 2022;
originally announced July 2022.
-
CPFN: Cascaded Primitive Fitting Networks for High-Resolution Point Clouds
Authors:
Eric-Tuan Lê,
Minhyuk Sung,
Duygu Ceylan,
Radomir Mech,
Tamy Boubekeur,
Niloy J. Mitra
Abstract:
Representing human-made objects as a collection of base primitives has a long history in computer vision and reverse engineering. In the case of high-resolution point cloud scans, the challenge is to be able to detect both large primitives as well as those explaining the detailed parts. While the classical RANSAC approach requires case-specific parameter tuning, state-of-the-art networks are limit…
▽ More
Representing human-made objects as a collection of base primitives has a long history in computer vision and reverse engineering. In the case of high-resolution point cloud scans, the challenge is to be able to detect both large primitives as well as those explaining the detailed parts. While the classical RANSAC approach requires case-specific parameter tuning, state-of-the-art networks are limited by memory consumption of their backbone modules such as PointNet++, and hence fail to detect the fine-scale primitives. We present Cascaded Primitive Fitting Networks (CPFN) that relies on an adaptive patch sampling network to assemble detection results of global and local primitive detection networks. As a key enabler, we present a merging formulation that dynamically aggregates the primitives across global and local scales. Our evaluation demonstrates that CPFN improves the state-of-the-art SPFN performance by 13-14% on high-resolution point cloud datasets and specifically improves the detection of fine-scale primitives by 20-22%.
△ Less
Submitted 6 September, 2021; v1 submitted 31 August, 2021;
originally announced September 2021.
-
Learning Generative Models of Shape Handles
Authors:
Matheus Gadelha,
Giorgio Gori,
Duygu Ceylan,
Radomir Mech,
Nathan Carr,
Tamy Boubekeur,
Rui Wang,
Subhransu Maji
Abstract:
We present a generative model to synthesize 3D shapes as sets of handles -- lightweight proxies that approximate the original 3D shape -- for applications in interactive editing, shape parsing, and building compact 3D representations. Our model can generate handle sets with varying cardinality and different types of handles (Figure 1). Key to our approach is a deep architecture that predicts both…
▽ More
We present a generative model to synthesize 3D shapes as sets of handles -- lightweight proxies that approximate the original 3D shape -- for applications in interactive editing, shape parsing, and building compact 3D representations. Our model can generate handle sets with varying cardinality and different types of handles (Figure 1). Key to our approach is a deep architecture that predicts both the parameters and existence of shape handles, and a novel similarity measure that can easily accommodate different types of handles, such as cuboids or sphere-meshes. We leverage the recent advances in semantic 3D annotation as well as automatic shape summarizing techniques to supervise our approach. We show that the resulting shape representations are intuitive and achieve superior quality than previous state-of-the-art. Finally, we demonstrate how our method can be used in applications such as interactive shape editing, completion, and interpolation, leveraging the latent space learned by our model to guide these tasks. Project page: http://mgadelha.me/shapehandles.
△ Less
Submitted 6 April, 2020;
originally announced April 2020.
-
Geometric Proxies for Live RGB-D Stream Enhancement and Consolidation
Authors:
Adrien Kaiser,
José Alonso Ybanez Zepeda,
Tamy Boubekeur
Abstract:
We propose a geometric superstructure for unified real-time processing of RGB-D data. Modern RGB-D sensors are widely used for indoor 3D capture, with applications ranging from modeling to robotics, through augmented reality. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. Our approach consists in genera…
▽ More
We propose a geometric superstructure for unified real-time processing of RGB-D data. Modern RGB-D sensors are widely used for indoor 3D capture, with applications ranging from modeling to robotics, through augmented reality. Nevertheless, their use is limited by their low resolution, with frames often corrupted with noise, missing data and temporal inconsistencies. Our approach consists in generating and updating through time a single set of compact local statistics parameterized over detected geometric proxies, which are fed from raw RGB-D data. Our proxies provide several processing primitives, which improve the quality of the RGB-D stream on the fly or lighten further operations. Experimental results confirm that our lightweight analysis framework copes well with embedded execution as well as moderate memory and computational capabilities compared to state-of-the-art methods. Processing RGB-D data with our proxies allows noise and temporal flickering removal, hole filling and resampling. As a substitute of the observed scene, our proxies can additionally be applied to compression and scene reconstruction. We present experiments performed with our framework in indoor scenes of different natures within a recent open RGB-D dataset.
△ Less
Submitted 21 January, 2020;
originally announced January 2020.
-
Plane Pair Matching for Efficient 3D View Registration
Authors:
Adrien Kaiser,
José Alonso Ybanez Zepeda,
Tamy Boubekeur
Abstract:
We present a novel method to estimate the motion matrix between overlap** pairs of 3D views in the context of indoor scenes. We use the Manhattan world assumption to introduce lightweight geometric constraints under the form of planes into the problem, which reduces complexity by taking into account the structure of the scene. In particular, we define a stochastic framework to categorize planes…
▽ More
We present a novel method to estimate the motion matrix between overlap** pairs of 3D views in the context of indoor scenes. We use the Manhattan world assumption to introduce lightweight geometric constraints under the form of planes into the problem, which reduces complexity by taking into account the structure of the scene. In particular, we define a stochastic framework to categorize planes as vertical or horizontal and parallel or non-parallel. We leverage this classification to match pairs of planes in overlap** views with point-of-view agnostic structural metrics. We propose to split the motion computation using the classification and estimate separately the rotation and translation of the sensor, using a quadric minimizer. We validate our approach on a toy example and present quantitative experiments on a public RGB-D dataset, comparing against recent state-of-the-art methods. Our evaluation shows that planar constraints only add low computational overhead while improving results in precision when applied after a prior coarse estimate. We conclude by giving hints towards extensions and improvements of current results.
△ Less
Submitted 20 January, 2020;
originally announced January 2020.
-
Real-time Deformation with Coupled Cages and Skeletons
Authors:
Fabrizio Corda,
Jean-Marc Thiery,
Marco Livesu,
Enrico Puppo,
Tamy Boubekeur,
Riccardo Scateni
Abstract:
Skeleton-based and cage-based deformation techniques represent the two most popular approaches to control real-time deformations of digital shapes and are, to a vast extent, complementary to one another. Despite their complementary roles, high-end modeling packages do not allow for seamless integration of such control structures, thus inducing a considerable burden on the user to maintain them syn…
▽ More
Skeleton-based and cage-based deformation techniques represent the two most popular approaches to control real-time deformations of digital shapes and are, to a vast extent, complementary to one another. Despite their complementary roles, high-end modeling packages do not allow for seamless integration of such control structures, thus inducing a considerable burden on the user to maintain them synchronized. In this paper, we propose a framework that seamlessly combines rigging skeletons and deformation cages, granting artists with a real-time deformation system that operates using any smooth combination of the two approaches. By coupling the deformation spaces of cages and skeletons, we access a much larger space, containing poses that are impossible to obtain by acting solely on a skeleton or a cage. Our method is oblivious to the specific techniques used to perform skinning and cage-based deformation, securing it compatible with pre-existing tools. We demonstrate the usefulness of our hybrid approach on a variety of examples.
△ Less
Submitted 6 September, 2019;
originally announced September 2019.
-
Jacobians and Hessians of Mean Value Coordinates for Closed Triangular Meshes
Authors:
Jean-Marc Thiery,
Julien Tierny,
Tamy Boubekeur
Abstract:
In this technical note, we present the formulae of the derivatives of the Mean Value Coordinates based transformations, using an enclosing triangle mesh, acting as a cage for the deformation of an interior object.
In this technical note, we present the formulae of the derivatives of the Mean Value Coordinates based transformations, using an enclosing triangle mesh, acting as a cage for the deformation of an interior object.
△ Less
Submitted 24 October, 2011; v1 submitted 9 September, 2011;
originally announced September 2011.
